Control and navigation systems, pose optimization, mapping, and localization techniques

ABSTRACT

A navigation program for an autonomous vehicle, the navigation program configured to: receive an initial model of an object to be inspected by the autonomous vehicle; identify an inspection target associated with the initial model of the object; and determine an inspection location for the autonomous vehicle from which inspection target is inspectable by an inspection system of the autonomous vehicle, wherein the initial model includes one or more convex shapes representing the object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is the U.S. National Stage Application of InternationalApplication No. PCT/GB2019/052521, filed Sep. 10, 2019, which claimspriority to United Kingdom Patent Application Serial No. 1902475.1,filed Feb. 22, 2019, Greek Patent Application Serial No. 20190100076,filed Feb. 13, 2019, United Kingdom Patent Application Serial No.1815864.2, filed Sep. 28, 2018, and Greek Patent Application Serial No.20180100410, filed Sep. 10, 2018, which are hereby incorporated byreference in their entirety.

TECHNICAL FIELD

Embodiments of the present invention relate to control and navigationsystems for autonomous vehicles and, in particular embodiments, forunmanned autonomous vehicles. Embodiments also relate to poseoptimization, mapping, and localization techniques.

BACKGROUND

With the decrease in the cost of sensor technology and computing power,among other things, there has been a great deal of recent development inrelation to autonomous vehicles, including unmanned autonomous vehicles(which are colloquially referred to as “drones”).

Such vehicles may be controlled by a human operator located remotelyfrom the drone and that control may be partially automated. For example,in the case of an aerial vehicle, one or more of the flight controls maybe autonomous such that the role of the operator is simplified.Nevertheless, there is a desire to provide fully autonomous drones whichare capable of performing tasks substantially without operatorintervention.

Effective substantially fully autonomous drones will reduce therequirement for a skilled operator (making the tasks those dronescomplete simpler and less expensive), reduce the risk of operator-error,and help to ensure repeatability along with a systematic approach tocertain tasks.

Some tasks in which drones are already being used include the inspectionof objects and structures. In particular, aerial drones have shown greatpotential in this regard—reducing the need for human inspections to takeplace (which can be extremely difficult in relation to some objects andstructures).

Increasing the degree of automation of drones performing such tasks isdifficult. In particular, the drone must be navigated within areal-world environment in which objects are not always in a known,and/or predetermined location and in which the drone (and other objects)are constantly exposed to changing environmental conditions.

There is a need, therefore, to provide local control and navigationsystems for autonomous vehicles, and/or better control and navigationsystems for autonomous vehicles.

Predicting Out-of-View Feature Points for Model-Based Camera PoseEstimation, by Oliver Moolan-Feroze, and Andrew Calway teaches aframework that uses deep learning to predict object feature points thatare out-of-view in the input image. This system was developed with theapplication of model-based tracking in mind, particularly in the case ofautonomous inspection robots, where only partial views of the object areavailable. Out-of-view prediction is enabled by applying scaling to thefeature point labels during network training. This is combined with arecurrent neural network architecture designed to provide the finalprediction layers with rich feature information from across the spatialextent of the input image.

BRIEF SUMMARY

Accordingly, aspects of the present invention are recited below and inthe appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are described, by way of example only, withreference to the accompanying drawings, in which:

FIG. 1 shows an example autonomous vehicle of some embodiments;

FIG. 2 shows a schematic view of an autonomous vehicle of someembodiments;

FIG. 3 shows a schematic view of a base station of some embodiments;

FIG. 4 shows a schematic view of an operator interface device of someembodiments;

FIG. 5 shows an example system of some embodiments;

FIG. 6 shows inspection targets in relation to a wind turbine;

FIG. 7 shows inspection targets and inspection locations with respect toa wind turbine;

FIG. 8 shows a point cloud misalignment;

FIG. 9 shows a state flow according to some embodiments;

FIG. 10 shows an autonomous vehicle performing an inspection operationin relation to an object;

FIG. 11 shows an autonomous vehicle performing an inspection operationin relation to an object;

FIG. 12 shows a flow diagram of a process of generating and combiningin-flight models according to some embodiments;

FIG. 13 shows a flow diagram of a process of generating and combiningin-flight models according to some embodiments;

FIG. 14 shows a model including feature lines overlaid on an image of awind turbine;

FIG. 15 shows a model including feature points overlaid on an image of awind turbine;

FIG. 16 shows a CNN architecture of some embodiments; and

FIG. 17 shows images from an autonomous vehicle including a projectionof a model onto the image, with the model shown both before and afteroptimization.

DETAILED DESCRIPTION

Embodiments of the present invention may include an autonomous vehicle1. The autonomous vehicle 1 could take a number of different forms andmany of the embodiments are described with reference to the autonomousvehicle 1 being an aerial vehicle (i.e., an aircraft). However, theautonomous vehicle 1 could be a land vehicle (configured for move acrossa ground surface), or a watercraft (configured to move through water),or an amphibious vehicle, for example.

The autonomous vehicle 1 may be a multirotor helicopter 1 andembodiments will be described, for convenience, in relation to amultirotor helicopter 1 (and, in particular, a quadcopter)—see FIG. 1 ,for example.

The autonomous vehicle 1 may include a frame 11 (i.e., a chassis)—seeFIG. 1 , for example. The frame 11 may be configured to support a drivesystem 12 of the autonomous vehicle 1—see FIG. 2 , for example. Thedrive system 12 is configured to drive movement of the autonomousvehicle 1. Accordingly, the drive system 12 may include one or moremotors 121 and/or one or more engines 122. The one or more motors 121may each be an electrical motor and the one or more engines 122 may eachbe an internal combustion engine. The drive system 12 may include one ormore gearboxes or other mechanical transmission systems.

The drive system 12 may be configured to drive rotational movement, forexample, of one or more drive elements 13—see FIGS. 1 and 2 , forexample—of the autonomous vehicle 1 with respect to the frame 11 aboutrespective rotational axes (via a gearbox or other mechanicaltransmission system, or otherwise). The or each drive element 13, whenso driven, is configured to drive movement of the autonomous vehicle1—e.g., through the air, along the ground, through the water, etc.

The or each drive element 13 may include a respective propeller, rotoror wheel, for example. In the case of a rotorcraft, for example, the oreach drive element 13 may include a rotor. The or each drive element 13may be configured, on rotation, to cause movement of the autonomousvehicle 1 in at least one direction.

As will be appreciated, depending on the nature of the autonomousvehicle 1 relatively complex arrangements of drive elements 13 may beprovided. This is especially true of rotorcraft—of which there arenumerous different known configurations.

In some embodiments, the drive system 12 may include one or more otheractuators which control or drive movement of the autonomous vehicle 1.For example, the drive system 12 may include one or more steeringmechanisms which are configured to move one or more steerable elements(such as wheels or a rudder) of the autonomous vehicle 1. The drivesystem 12 may include one or flight control actuators which areconfigured to move one or more flight control surfaces of the autonomousvehicle 1 (similar control actuators and surfaces may be provided forother types of craft, such as watercraft and “flight” control actuatorsand surfaces should be construed accordingly). In general terms, theseare other examples of drive elements 13 and the actuators may beelectrically or hydraulically driven for example.

In the case of a quadcopter as the autonomous vehicle 1, for example,there may be four drive elements 13 which each comprise a rotor.However, as in the depicted example, a quadcopter may also use a coaxialdrive element 13 configuration such that there is a total of eight driveelements 13 arranged in four co-axial pairs. This, as will beappreciated, is just one example configuration.

The precise configuration of the drive elements 13 with respect to theframe 11 will depend on the nature of the autonomous vehicle 1 and theintended movements of the autonomous vehicle 1. In some embodiments inwhich the autonomous vehicle 1 is a multirotor helicopter, theautonomous vehicle 1 includes a plurality of drive elements 13 which arearranged with rotational axes which are generally vertical in normaloperation. In fixed wing aircraft and/or boats and/or land vehicles, theor each drive element 13 have a rotational axis which is substantiallyhorizontal in normal use.

In some embodiments, the frame 11 includes a number of arms 111 and eacharm 111 may carry one or more of the drive elements 13. The frame 11 mayinclude a main structure 112 which may be located generally centrallywithin the autonomous vehicle 1 in some embodiments. The frame 11 mayinclude one or more legs 113 which are configured to support the mainstructure 112 with respect to a ground surface. In some embodiments, theone or more legs 113 may not be provided and the one or more arms 111with their drive elements 13 may provide this support (e.g., inautonomous vehicles 1 which are configured to move over the groundsurface with a drive element 13 in the form of a wheel).

In some embodiments, the drive system 12 is mounted primarily on thecentral structure 112 and mechanical power may be transmitted to the oreach drive element 13 using a mechanical transmission (as describedherein).

The drive system 12 is coupled to a power source 14 of the autonomousvehicle 1. In the case of a drive system 12 including one or more motors121 (i.e., electrical motors), the power source 14 may include abattery, for example, or another form of electrical power storagedevice. In the case in which the drive system 12 includes one or moreengines 122 (e.g., internal combustion engines), then the power source14 may include a tank of fuel (which may be a hydrocarbon fuel such asgasoline or natural gas or liquefied petroleum gas or the like). Thepower source 14 may, in some embodiments, include a power generator(such as a fuel cell or solar panel or the like) which is configured togenerate electrical power rather than to store electrical power.

As will be appreciated, therefore, the flow of electrical power and/orfuel to the or each motor 121 and/or engine 122 may be controlled by thedrive system 12. Accordingly, the drive system 12 can be controlled toalter one or more operational characteristics of the or each motor orengine 121,122—such as the speed at which the motor and/or engine121,122 drives the associated drive element 13. Likewise, the drivesystem 12 may control the operation of one or more other actuators asdescribed herein.

The power source 14 may be mounted on the frame 11 (e.g., the centralstructure 112 thereof).

The drive system 12 is communicatively coupled to a flight controlsystem 15 of the autonomous vehicle 1. The flight control system 15 isconfigured to control the operation of the drive system 12 in order tocontrol one or more aspects of the operation (e.g., movement) of theautonomous vehicle 1. Whilst this is labelled as a flight control system15 it will be appreciated that this label is primarily assigned withaircraft in mind. For embodiments in which the autonomous vehicle 1 isnot an aircraft, then a different label may be applied to the flightcontrol system 15—for example, a “drive control system” (which should beconstrued as encompassing the flight control system 15).

The flight control system 15 may take the form of a suitably programmedflight control computer (again, this label may be changed to drivecontrol computer in embodiments in which the autonomous vehicle 1 is notan aircraft).

The flight control system 15 may be mounted on the frame 11 (e.g., thecentral structure 112 thereof).

As mentioned above, the flight control system 15 is configured tocontrol the operation of the drive system 12 and, with this in mind, theflight control system 15 is communicatively coupled to a navigationsystem 16 of the autonomous vehicle 1. The navigation system 16 isconfigured to provide instructions to the flight control system 15 forcontrolling movement of the autonomous vehicle 1 in accordance with anavigational plan (i.e., a flight plan defining a flight path) stored inand/or generated by the navigation system 16. In some embodiments, theflight control system 15 is configured to control the operation of thedrive system 12 to control the orientation and/or position of theautonomous vehicle 1. In some embodiments, in which the autonomousvehicle 1 is capable of movement in one or more of an up-down degree offreedom, a left-right degree of freedom, a forward-backward degree offreedom, a pitch degree of freedom, a yaw degree of freedom, and a rolldegree of freedom, the flight control system 15 is configured to controlmovement in that or those degree(s) of freedom of movement.

In other words, the flight control system 15 is primarily concerned withcontrolling the autonomous vehicle 1 to achieve movements which arespecified by the navigation system 16. The navigation system 16determines where the autonomous vehicle should travel and the flightcontrol system 15 is configured to control the autonomous vehicle 1 tomove in the desired manner based on the instruction from the navigationsystem 16.

The navigation system 16 may take the form of a suitably programmednavigation computer and this computer might be the same computer as theflight control computer—the computer running suitable programs such thatthe computer performs both roles.

In some embodiments, the navigation system 16 may be mounted on theframe 11 (e.g., the central structure 112 thereof). However, in someembodiments, the navigation system 16 may be remote from the autonomousvehicle 1—in other words, the navigation system 16 may becommunicatively coupled to the flight control system 15 (i.e., to theautonomous vehicle 1) but may not travel with the autonomous vehicle 1.In some embodiments, the navigation system 16 is a distributed systemsuch that some parts of the system are carried by the autonomous vehicle1 and other parts are remotely located with respect to the autonomousvehicle 1.

The autonomous vehicle 1 includes a sensor system 17. The sensor system17 is configured to sense one or more aspects of the environment of theautonomous vehicle 1 and to provide these one or more sensed aspects tothe navigation system 16 and/or the flight control system 15 to allowthose systems 15/16 to perform their respective functions.

The sensor system 17 may, therefore, include one more sensors 171 eachof which is configured to sense a different aspect of the environment ofthe autonomous vehicle 1. These one or more sensors 171 may be referredto as a sensor suite of the autonomous vehicle 1. The one or moresensors 171 may include, for example, one or more: accelerometers,gyroscopic sensors, magnetometers, radio navigation systems (such as asatellite-based radio navigation system (e.g., using the GlobalPositioning Satellite system, GLONASS, Galileo, Beidou, IRNSS, orQZSS)), and/or the like.

The one or more sensors 171 generate sensor outputs which are passed tothe flight control system 15 and/or the navigation system 16. Thesesystems 15/16 use the sensor outputs in order to control one or moreaspects of the operation of the autonomous vehicle 1 based at least inpart on the sensor outputs.

The sensor system 17 may be mounted on the frame 11 (e.g., the centralstructure 112 thereof in some embodiments). In some embodiments,however, the one or more sensors 171 of the sensor system 17 may beotherwise located on or around the autonomous vehicle 1.

In some embodiments, at least one of the one or more sensors 171 is atleast partially located remotely from the autonomous vehicle 1 (i.e., isnot carried for movement with the autonomous vehicle 1). For example,the one or more sensors 171 may include a differential radio navigationsystem (such as a differential GPS system). In such embodiments, the oneor more sensors 171 may include a radio navigation system sensor whichis carried by the autonomous vehicle 1 and which is communicativelycoupled (e.g., via a wireless communication channel—see thecommunication system 18) to a base station 2 (see FIGS. 3 and 5 , forexample) which is configured to generate a differential signal. Theradio navigation system sensor may use the differential signal tocorrect a locally determined location generated by the radio navigationsystem sensor (wherein “locally” in this instance means determined bythe radio navigation system sensor carried by the autonomous vehicle 1).In such examples, the sensor 171 may be viewed as a distributed sensor(such that the at least part of the base station 2 forms part of thatsensor) or as a sensor 171 which receives a sensor output from anothersensor to generate a modified or corrected sensor output. In someembodiments, the sensor may use real time kinematic positioning.

The sensor system 17 may include a ranging sensor as one of the one ormore sensors 171. The ranging sensor is configured to determine a range(i.e., a distance) of another object from the ranging sensor (and sofrom the autonomous vehicle 1 to which the ranging sensor is mounted).The range may be output as sensor data, for example. The ranging sensormay include, for example, a light detection and ranging sensor (i.e., aLiDAR sensor). The ranging sensor may include, for example, a soundnavigation and ranging sensor (i.e., a sonar sensor or other acousticsensor, such as an ultrasound ranging sensor). The ranging sensor mayinclude, for example, a radio detection and ranging sensor (i.e., aradar sensor). The ranging sensor may include, for example, a rangeimaging sensor (which may use, for example, stereoscopic images (e.g.,of the visible light) to determine range). The ranging sensor mayinclude a 1-D ranging sensor and/or a 2-D ranging sensor and/or a 3-Dranging sensor.

The autonomous vehicle 1 may include an inspection system 19 which isconfigured to inspect another object or volume (i.e., an object otherthan the autonomous vehicle 1). The inspection system 19 may beconfigured to output inspection data which is indicative of one or morecharacteristics of the other object. The inspection system 19,therefore, includes one or more inspection sensors 191. In someembodiments, the inspection system 19 includes a camera 191 a (as aninspection sensor 191) which is configured to capture one or more imagesof the other object. The camera 191 a may be configured to capture lightin the visible light spectrum but additionally or alternatively may beconfigured to capture infrared and/or ultraviolent light images of theobject. As will be appreciated, therefore, the camera 191 a may comprisemore than one camera sub-system—e.g., with each camera sub-systemconfigured to capture an image in a different spectrum or from adifferent position relative to at least one other camera sub-system(e.g., to obtain a stereoscopic image).

In some embodiments, the camera 191 a or camera sub-system may beconfigured to capture an image of a particular polarization of light(which may be particularly useful for inspecting some other objects 5 orvolumes).

In some embodiments, the inspection system 19 includes one or moresensors 191 which are not a camera 191 a. For example, the inspectionsystem 19 may include an air quality sensor 191 b which may beconfigured to detect one or more contaminants in the air (e.g.,particulate matter). The inspection system 19 may include a radiationsensor 191 c which may be configured to detect one or more contaminantsin the air (e.g., particulate matter). The inspection system 19 mayinclude an acoustic sensor as one of the one or more inspection sensors191 which may be, for example, an ultrasound sensor. The one or moreinspection sensors 19 may include a non-destructive testing sensor whichmay be an ultrasonic non-destructive testing sensor.

In some embodiments, the inspection system 19 may include a projectorwhich is configured to emit (or project) a signal towards the object 5.The signal may be an acoustic or electromagnetic signal (e.g., light),for example. The inspection system 19 may include, in such embodiments,a receiver which is configured to receive at least part of the emittedsignal—e.g., after reflection from the object 5. One or morecharacteristics of the object 5 may be determined by the inspectionsystem 19 using the emitted signal and a received portion of the emittedsignal (e.g., after reflection from the object 5). The projector may beconfigured to project structured light and the inspection system 19 mayinclude a so-called structured light sensor system or structured-light3D scanner.

The or each inspection sensor 191 may be mounted with respect to theframe 11 of the autonomous vehicle 1 using a respective mounting device193. In some embodiments, there may be multiple inspection sensors 191mounted with respect to the frame 11 by the same mounting device 193.

The mounting device 193 (or devices 193) used to secure the or eachinspection sensor 191 may take a number of different forms. In someembodiments, the mounting device 193 is configured to provide a fixedpositional arrangement with respect to the frame 11. As such, themounting device 193 may include, for example, a substantially rigidbracket configured to be attached to the frame 11 (directly orindirectly) and to at least one of the one or more inspection sensors191. The mounting device 193 may include an armature which is configuredto move with respect to at least part of the frame 11 so that theposition of the inspection sensor 191 is moveable with respect to theframe 11—this movement may be driven by an actuator under the control ofan inspection system computer 192. In some embodiments, the mountingdevice 193 is configured to provide a substantially fixed orientationbetween the inspection sensor 191 and one or more other objects (such asthe other object 5). In such embodiments, the mounting device 193 mayinclude a gimbal mechanism which may have one to three degrees offreedom of movement. The provision of a mounting device 193 in the formof a gimbal mechanism may be used in relation to an inspection sensor191 which includes the camera 191 a—to assist with stabilizing thecamera 191 a in order to improve the quality of an image captured by thecamera 191 a.

The inspection system 19 may include the inspection system computer 192(which may be the same computer as the navigation computer and/or theflight control computer—the computer running suitable programs such thatthe computer performs all of these roles). The inspection systemcomputer 192 may be configured to control one or more aspects of theoperation of the or each inspection sensor 191—for example, when animage is to be captured by the camera 191 a or a sample taken by the airquality sensor 191 b or radiation sensor 191 c, and/or one or moreparameters of the operation of the or each inspection sensor 191 (suchas, in relation to the camera 191 a, focus, shutter speed, aperturesize, and the like).

In some embodiments, the inspection system computer 192 may beconfigured to receive the inspection data from the or each inspectionsensor 191 and may store that inspection data for subsequenttransmission or release. The inspection system computer 192 may beconfigured to determine when and how the inspection data is to behandled (e.g., transmitted or deleted). The inspection system computer192 may, therefore, include a storage medium 192 a which is configuredto store the inspection data.

The inspection system 19 may be mounted on the frame 11 (e.g., thecentral structure 112 thereof).

In some embodiments, the autonomous vehicle 1 may include a visualguidance system 20 which is configured to capture one or more images foruse by an operator to assist in operating and/or controlling theautonomous vehicle 1—see herein elsewhere. The visual guidance system 20is configured, therefore, to capture an image (e.g., using a camera ofthe visual guidance system 20) and to generate guidance data whichrepresents that image (e.g., such that at least part of the image can bereconstructed based on the guidance data).

The visual guidance system 20 may be mounted on the frame 11 (e.g., thecentral structure 112 thereof) or may be mounted on one of the arms 111if provided.

The autonomous vehicle 1 may include a communication system 18. Thecommunication system 18 may be communicatively coupled to one or more ofthe drive system 12, the power source 14, the flight control system 15,the navigation system 16, the sensor system 17, the inspection system19, and the visual guidance system 20. In particular, the communicationsystem 18 may be communicatively coupled to one or more of the flightcontrol computer, the navigation computer, and the inspection systemcomputer 192 (which may be the same computer, suitably programmed).

The communication system 18 is configured to transmit data from theautonomous vehicle 1 to a remote location (i.e., an element which is notcarried by the autonomous vehicle 1). The communication system 18 isalso configured to receive data from a remote location for use by theautonomous vehicle 1.

An example of data which may be received by the communication system 18includes the differential signal—as described herein. Other data whichmay be received by the communication system 18 include one or more of:other sensor data (i.e. data from one or more sensors which are locatedremotely from the autonomous vehicle 1 (e.g. in the base station 2)),control information (such as flight control information for use by theflight control system 15 in the control of the operation of the drivesystem 12 (which may be direct flight control information such that thenavigation system 16 is effectively circumvented), and navigationinformation (for use by the navigation system 16).

The communication system 18 may include one or more communicationinterfaces 181 and each communication interface 181 may be configured totransmit and/or receive data using a different communication channeland/or protocol.

One or more of the communication interfaces 181 may include a wirelesscommunication interface (such as a radio frequency communicationinterface). The one or more communication interfaces 181 may include awireless long range communication interface and a wireless short rangecommunication interface. The wireless short range communicationinterface may be configured to send and/or receive data over arelatively short range and may be used for control and/or navigationinformation, for example (which may include the guidance data). Thewireless long range communication interface may be configured to sendand/or received data over a relatively longer range and may be used forless time critical data—such as a general telemetry information, orinspection data (as is described elsewhere).

In some embodiments, therefore, the one or more communication interfaces181 may include a plurality of wireless communication interfaces. Theone or more communication interfaces 181 may include one or more of aWiFi communication interface (or other communication interface operatingat around 2.4 GHz, 3.6 GHz, 5 GHz, and/or 60 GHz), a communicationinterface operating at around 433 MHz, 915 MHz, and/or 2400 MHz, and/ora communication interface operating at around 315 MHz and/or 868 MHz,and/or a cellular communication network interface (which may use abroadband cellular telephone network communication system).

The communication system 18 may be mounted on the frame 11 (e.g., thecentral structure 112 thereof) or may be mounted on one of the arms 111if provided.

The power source 14 may be configured to provide electrical power to oneor more of the communication system 18, the drive system 12, the flightcontrol system 15, the navigation system 16, the sensor system 17, theinspection system 19, and the visual guidance system 20.

The base station 2—see FIG. 3 , for example—may be configured tocommunicate with the autonomous vehicle 1 (and vice versa) via thecommunication system 18 of the autonomous vehicle 1. In someembodiments, the base station 2 is not provided. In some embodiments,the base station 2 is a unit which is configured to be placed generallywithin an operating area of the autonomous vehicle 1 (but which may beremoved and transported to a different operating area for the autonomousvehicle 1 if required). In some instances, the base station 2 is a unitwhich has a substantially fixed geographical location and cannot readilybe removed and transported to a new location. In some instances, thebase station 2 is or includes a mobile computing device such as a mobiletelephone, laptop, or tablet computer.

The base station 2 may include a base station communication system 21which is configured to be communicatively coupled to the communicationsystem 18 of the autonomous vehicle 1. The base station communicationsystem 21 may include one or more communication interfaces 211 and eachcommunication interface 211 may be configured to transmit and/or receivedata using a different communication channel and/or protocol.

One or more of the communication interfaces 211 may include a wirelesscommunication interface (such as a radio frequency communicationinterface). The one or more communication interfaces 211 may include awireless long range communication interface and a wireless short rangecommunication interface. The wireless short range communicationinterface may be configured to send and/or receive data over arelatively short range and may be used for communication with theautonomous vehicle 1 for example. The wireless long range communicationinterface may be configured to send and/or received data over arelatively longer range and may be used for connection to a remoteserver 3 and/or the Internet.

In some embodiments, therefore, the one or more communication interfaces211 may include a plurality of wireless communication interfaces. Theone or more communication interfaces 211 may include one or more of aWiFi communication interface (or other communication interface operatingat around 2.4 GHz, 3.6 GHz, 5 GHz, and/or 60 GHz), a communicationinterface operating at around 433 MHz, 915 MHz, and/or 2400 MHz, and/ora communication interface operating at around 315 MHz and/or 868 MHz,and/or a cellular communication network interface (which may use abroadband cellular telephone network communication system).

The base station 2 may include a base station computer 22 which isprogrammed to perform one or more tasks as described herein. The basestation computer 22 may be communicatively coupled to the base stationcommunication system 21 to effect the communication of data to and fromthe base station 2 and, in particular, the base station computer 22.

The base station computer 22 may include a storage medium 221, the roleof which is described herein.

The base station 2 may include a position sensor system 23 whichincludes at least one position sensor 231 which is configured todetermine a geographical location of the base station 2. The at leastone position sensor 231 may include a radio navigation system sensor(such as a satellite-based radio navigation system sensor (e.g., usingthe Global Positioning Satellite system, GLONASS, Galileo, Beidou,IRNSS, or QZSS).

The position sensor system 23 may be communicatively coupled to the basestation computer 22 and/or to the base station communication system 21.The position sensor system 23 may, therefore, be configured to send basestation position data to the base station computer 22 and/or to the basestation communication system 21 for transmittal onwards to one or moreother devices (such as the autonomous vehicle 1 and/or the remote server3).

In some embodiments, the base station 2 (e.g., using the position sensorsystem 23) may be configured to determine a differential signal which isoutput as at least part of the base station position data (e.g., for useby the autonomous vehicle 1 as described herein).

The base station 2 may include a base station power source 24. In someembodiments, the base station power source 24 may be a source ofelectrical power and the base station power source 24 may be coupled toone or more of the base station communication system 21, the basestation computer 22, and the position sensor system 23, to provideelectrical power thereto.

In some embodiments, the base station power source 24 includes one ormore of a battery, a connection to a mains electrical power supply, anelectrical generator (which may include an internal combustion engine),and an alternator of a vehicle (such as a support vehicle used totransport the autonomous vehicle 1 to the operating area).

A combination of the autonomous vehicle 1 and the base station 2 may bereferred to as an autonomous vehicle system 100 (see FIG. 5 , forexample), for example. As described elsewhere, however, the base station2 need not be provided in some embodiments.

In some embodiments, the autonomous vehicle system 100 may include anoperator interface device 4 (see FIG. 4 , for example). In suchembodiments, the autonomous vehicle system 100 may not include the basestation 2—such that the autonomous vehicle system 100 includes theautonomous vehicle system 100 and the operator interface device 4. Insome embodiments, the autonomous vehicle system 100 includes theautonomous vehicle 1, the base station 2, and the operator interfacedevice 4. In some embodiments the autonomous vehicle system 100 mayinclude the remote server 3 too.

In some embodiments, the operator interface device 4 includes acomputing device 41 which is programmed to provide an operator with aninterface 42 using which the operator can control one or more aspects ofthe operation of the autonomous vehicle 1.

The computing device 41 may be a mobile computing device such as amobile telephone, a tablet, or a laptop. As such the computing device 41may include a power source 43 such as a battery.

The operator interface device 4 includes an interface devicecommunication system 44 which is configured to communicate with the basestation 2 (e.g., with the base station communication system 21) and/orthe autonomous vehicle 1 (e.g., the communication system 18 thereof).Accordingly, the interface device communication system 44 may be a wiredor wireless communication system which may be configured to use acommunication interface corresponding with one of the communicationinterfaces 181,211 described herein. In some embodiments, the interfacedevice communication system 44 includes a plurality of suchcommunication interfaces and uses different such interfaces tocommunicate with the base station 2 and the autonomous vehicle 1.

In some embodiments, the base station 2 and the operator interfacedevice 4 are one and the same device 2,4.

The autonomous vehicle 1 (and the autonomous vehicle system 100) may beconfigured to inspect another object or volume 5. That other object orvolume 5 may be, for example, one or more of a wind turbine,photovoltaic cells, aircraft external structures (such as wings,undercarriage, engines, tail, and fuselage), a construction site, ahigh-rise building, a boat, a dam, a bridge, a power line, a pipeline, asea wall, a harbor wall, flood defenses, a railway line, a harbor (forair quality), a city (for air quality), a building, a tank, a nuclearreactor or associated parts of a nuclear power facility, a pile, a cable(undersee or overhead), and the like.

As will be appreciated, the one or more inspection sensors 191 which areprovided as part of the autonomous vehicle 1 may be selected inaccordance with the nature of the inspection task to be undertaken bythe autonomous vehicle 1.

Embodiments are described with specific reference to the inspection of awind turbine as the other object 5 (i.e., the subject of the inspectiontask); however, it will be appreciated that this description appliesequally to the inspection of other objections and volumes, such as thosementioned herein.

As will be apparent, the various components of the autonomous vehiclesystems 100 described herein communicate with each other using theirrespective communication systems 18,21,44. These communications may beover any of the wired or wireless communications channels described inrelation to these systems 18,21,44, for example.

In some embodiments, the base station 2, if provided, may act as a relaystation between the operator interface device 4 and the autonomousvehicle 1 such that communications between the operator interface device4 and the autonomous vehicle 1 are channeled through the base station2—which may act as a signal repeater and/or may receive thecommunications via one communication channel and relay them throughanother (different) communications channel.

The use of the base station 2 as a relay for communications may reducethe required transmission power of the communication systems 18,44 ofthe autonomous vehicle 1 and the operator interface device 4.

In some embodiments, the base station 2 may act as a relay forcommunications to the remote server 3—such that data sent from theautonomous vehicle 1 and/or the operator interface device 4 for theremote server 3 is first sent to the base station 2 which thenre-transmits the data to the remote server 3 (e.g., using a differentcommunication interface, such as the long range communicationinterface).

The operation of the autonomous vehicle 1, the base station 2, and theoperator interface device 4 (and the remote server 3) is controlled bythe one or more computer programs which are executed by the flightcontrol computer and/or the navigation computer and/or the inspectionsystem computer and/or the base station computer 22 and/or the computingdevice 41 and/or the remote server 3. As will be appreciated, theoperation of these computers may be combined and/or distributed asdesired to achieve the required control. The present descriptionpresents one or more arrangements of the computer programs but this isnot to be viewed as limiting on the disclosure as a whole.

The flight control computer is configured to execute a flight controlprogram. The flight control program is configured to receive flightcontrol instructions comprising desired movements of the autonomousvehicle 1 within a volume—i.e., the operating area. The flight controlprogram may be configured to receive inputs from the sensor system 17(e.g., from the one or more sensors 171) which provide the flightcontrol program with sensor data which enables the flight controlprogram to seek to control the autonomous vehicle 1 in accordance withthe flight control instructions. The flight control program may not, forexample, be configured to determine a desired movement of the autonomousvehicle 1 but, instead, receives the desired movements (in the form ofthe flight control instructions) and then seeks to achieve those desiredmovements. The flight control program may, therefore, be configured tocontrol the operation of the drive system 12 to seek to achieve thedesired movements and this may include controlling the movement of theor each drive element 13 or other actuator (which may be a form of driveelement 13 as described). As will be appreciated, the flight controlprogram may be substantially unique to a particular autonomous vehicle 1or type of autonomous vehicle 1, as the configurations of such vehiclescan vary greatly from one to another.

The computing device 41 of the operator interface device 4 may includean interface 42 through which the operator can generate flight controlinstructions for transmittal to the flight control program to controlthe operation of the autonomous vehicle 1 manually. These flight controlinstructions may be referred to as, for example, the direct flightcontrol information—because they are instructions which have not beengenerated by the navigation program. This interface 42 may form part ofan operator control program which is configured to be executed by thecomputing device 41 of the operator interface device 4.

Accordingly, the operator interface device 4 may be used for manualcontrol of the autonomous vehicle 1 if the need arises.

The navigation computer is configured to execute a navigation programwhich is configured to provide the one or more flight controlinstructions to the flight control program. The navigation computer mayalso receive inputs from the sensor system 17 (e.g., from the or eachsensor 171) and may also receive inputs from the inspection system 19and/or the visual guidance system 20.

It is the navigation program which determines a flight plan (i.e., adesired flight path) for the autonomous vehicle 1 in normal use, issuesthe flight control instructions with a view to achieving that flightpath, and which monitors adherence to that flight path (and takescorrective action as required).

The navigation program may be communicatively coupled to an initialsetup program which is used to provide the navigation program withinitial information (e.g., initial setup parameters) in order toestablish the flight plan.

The initial setup program may be configured, therefore, to receive oneor more initial setup parameters for the navigation program. The initialsetup program may be configured to be executed by the operator interfacedevice 4 and, in particular, by the computing device 41 thereof. Theoperator may, therefore, be able to enter one or more of the initialsetup parameters using the interface 42 of the operator interface device4. In some embodiments, one or more of these initial setup parametersmay be obtained from the remote server 3—having been previously enteredand then stored in the remote server 3 for future use. In someembodiments, the initial setup program may be executed by the basestation computer 22 and an interface may be served to the operatorinterface device 4 by the base station computer 2—to enable inputs fromthe operator to be received via the operator interface device 4.

More specifically, the initial setup program may receive one or moreinitial setup parameters which may include one or more of:

-   -   the type of object 5 to be inspected;    -   an approximate location of the object 5 (e.g., within 100 m of        the actual location or within 50 m of the actual location,        and/or with a location error of at least 1 m, or 5 m, or 10 m,        or 20 m, or 30 m);    -   a precise location of the object 5 (e.g., with a location error        of less than 5 m, or 1 m, and in any event with a location error        less than the location error of an approximate location of the        object 5);    -   a model of the object 5;    -   one or more physical characteristics of the object 5;    -   an approximate direction of the object 5 from the autonomous        vehicle 1; and    -   a position of one or more parts of the object 5.

In the example of the object 5 being a wind turbine, the type of object5 may be an indication that the object is a wind turbine. Suchinformation may be required if the initial setup program and/ornavigation program are configured for use in inspecting multipledifferent types of object (or a volume which will be considered anotherform of object). However, in some embodiments, the initial setup programand/or the navigation program are configured for use in inspecting onlywind turbines. Therefore, this information may already be providedwithin the initial setup program and/or the navigation program.

An approximate location of the object 5 may include an approximatelongitude and latitude of the object 5—in other words an approximategeographic location. This information may be useful in determining aninitial flight path of the autonomous vehicle 1 and/or for recording theinspection data against the location for future reference.

Similarly, the precise location of the object 5 may be a more precisegeographical location of the object 5. This information may not beavailable for all objects 5 and it may be that the approximategeographical location can be provided but the precise location cannot.Again, this information may be used in much the same manner as theapproximate location information above.

A model of the object 5 may, in the case of a wind turbine, be the model(e.g., type as opposed to a representation) of the wind turbine and mayinclude a manufacturer. Whilst there will be some variation betweenspecific instances of a particular model of an object, such as a windturbine, all models will likely share certain characteristics.Therefore, this information may be used in order to lookup—e.g., using adatabase of the initial setup program or the navigation program orobtained from the remote server 3—one or more characteristics of theobject 5. Such characteristics may include, in the case of a windturbine as an example, a height of the tower, a size of the nacelle, asize of the hub, a size of the blades, a number of blades, and the like.Again, this information is useful for determining the initial flightpath and for recording against a record for the inspection.

One or more physical characteristics of the object 5 may include thesame types of information which may be obtained by using the model ofthe object 5 and so may be manually entered if known, for example.

An approximate direction of the object 5 from the autonomous vehicle 1may be, for example, that the object 5 is to the north/south/east/westof the autonomous vehicle 1. As will be apparent from the presentdescription, as part of the initial flight path the autonomous vehicle 1moves to confirm the location of the object 5. An approximate directionof the object 5 from the autonomous vehicle 5 at the start of the flightof the autonomous vehicle 1 will help to ensure that the autonomousvehicle finds the object 5 relatively quickly and successfully (and/orthat the correct object 5 is found—which may be a particular issue inrelation to operation of the autonomous vehicle 1 in close proximity tomany similar objects 5 (such as on a wind farm)).

A position of one or more parts of the object 5 may be, for example, theposition of one or more blades of a wind turbine with respect to thetower of the wind turbine. Again, this may assist in the planning of theinitial flight path.

In some embodiments, the initial setup program is configured to presentto an operator a map or aerial photograph, via the operator interfacedevice 4 (e.g., the interface 42). The initial setup program may be ableto receive a user input (via the interface 42) of the current locationof the autonomous vehicle 1 with respect to the object 5 (which may beshown in the map or aerial photograph). The initial setup program may beconfigured to use this information to determine, for example, theapproximate location of the autonomous vehicle 1 and/or the approximatedirection of the object 5 from the autonomous vehicle 1.

The initial setup program may otherwise or additionally present theoperator—via the operator interface device 4 (e.g., the interface42)—with a form with one or more associated data entry fields throughwhich any or all of the above information may be entered.

In some embodiments, there may be a specific start location for theautonomous vehicle 1 with respect to the object 5—such as a launch pador other marker. The operator may, therefore, locate the autonomousvehicle 1 at the specific start location and then use the operatorinterface device 4 to provide an indication to the initial setup programthat the autonomous vehicle 1 is at the specific start location forinspecting that object 5.

In some embodiments, the operator may orient the autonomous vehicle 1such that the orientation of the autonomous vehicle 1 with respect tothe object 5 is predetermined—thus, providing the approximate directionof the object 5 with respect to the autonomous vehicle 1. The autonomousvehicle 1 may be provided with a visual indicator in this respect forthe operator to align with the object 5.

In some embodiments, the operator may be required to position theautonomous vehicle 1 at a predetermined orientation with respect to theobject 5. In the case of a wind turbine, for example, this may beoriented such that the autonomous vehicle is in front of the hub andblades (with respect to the tower). This may assist in providing theinitial setup program and/or navigation program with information aboutthe position of parts of the object 5 with respect to the autonomousvehicle 1.

With some or all of this initial data provided to the initial setupprogram, the initial setup program may pass this data to the navigationprogram. In some instances, the navigation program is executedsubstantially entirely by the navigation computer (which may be mountedon the frame 11 of the autonomous vehicle 1); however, in someembodiments at least part of that navigation program is executed by theoperator interface device 4 and/or the base station computer 22.

The navigation program may also receive one or more inputs from thesensor system 17 of the autonomous vehicle 1—e.g., sensor data from theone or more sensors 171. This sensor data may include, for example,location information determined by the one or more sensors 171 for thegeographical location of the autonomous vehicle 1 and/or its orientation(e.g., with respect to the earth's magnetic field using a sensor 171 inthe form of a magnetometer).

The navigation program uses some or all of the data provided by theinitial setup program (and/or predetermined information as a result ofdefined requirements for the operator to locate the autonomous vehicle 1with respect to the object 5, as discussed above) to determine anapproximate location of the object 5 from the autonomous vehicle 1 andto plan an initial flight path.

The navigation program may be configured to generate a visualrepresentation of the approximate location and configuration of theobject 5 and the autonomous vehicle 1 along with a visual representationof the initial flight path with respect thereto. This may be output tothe operator interface device 4 for the operator to view and confirm.

The initial flight path may be defined within a frame of reference ofthe navigation system which may be aligned with a global frame ofreference using, for example, geographical location information from theone or more sensors 171 (which may be refined using the base stationlocation data, for example).

The initial flight plan is a flight plan which moves the autonomousvehicle 1 to one or more inspection locations at which the inspectionsystem 19 of the autonomous vehicle 1 may be used to perform aninspection of the object 5. This may include, for example, capturing animage of at least part of the object 5 using the camera 191 a and/orusing one or more of the other inspection sensors 191 if provided.

The inspection locations are, therefore, locations which the autonomousvehicle 1 must reach. Moving between these locations, however, exposesthe autonomous vehicle to many potential hazards, for example. A primaryhazard is the risk of impacting the object 5. Therefore, there is needto define a safe flight path between the inspection location(s) and/orstart and end locations (at which the autonomous vehicle 1 starts andends its flight).

This may be achieved in some embodiments through the navigation programdefining one or more transition locations. The transition locations arelocations through which the autonomous vehicle 1 must pass.

The navigation program may also be configured to define a sequence ofinspection location(s) and transition location(s) to provide the flightpath and plan.

The flight plan may, therefore, comprise a sequence of locations (or“points”) which include the one or more inspection locations (or“points”) and one or more transition locations (or “points”). Thesequence also includes a start location (or “point”) and an end location(or “point”). The start and end locations may be one and the same andthis may be referred to as a home location—which may be a home locationwhich is specific to that flight in some embodiments, for example.

In some embodiments, a transition location is a location in a plane oron a locus which is defined as a safe plane or locus.

There may be an orientation of the autonomous vehicle 1 and/or theinspection system 19 with respect to the object 5 associated with eachinspection location and/or transition location within the flight plan—toensure that the inspection system 19 is correctly positioned to inspectthe object 5.

The autonomous vehicle 1 may, under the instruction from the navigationprogram, commence a start-up routine which includes one or more oftesting the sensor system 17, testing the communication system 18,testing the inspection system 19, and testing the drive system 12. Insome embodiments, the navigation program is configured to estimate, forexample, the power requirements in order to complete the initial flightpath. This may be compared to a power availability of the power source14 (e.g., a battery charge level) in order to determine whether theflight can be completed with the available power.

The autonomous vehicle 1 may, if the start-up routine completessuccessful, enter a ready state. The ready state may be indicated to theoperator via the operator interface device 4. If the start-up routinefails to complete successful—e.g., because one or more of the testsfails—then the autonomous vehicle 1 may remain in an unready state andthis may also be indicated to the operator via the operator interfacedevice 4 (e.g., along with an indication of test which failed).

With the autonomous vehicle 1 in the ready state the operator maytrigger operation of the autonomous vehicle 1 in accordance with theinitial flight plan via the operator interface device 4—e.g., throughthe interface 42.

In the example case of an aircraft as the autonomous vehicle 1,commencement of the operation of the operation of the vehicle 1 mayinclude a spin-up cycle (in which the or each drive element 13 is drivenup to a required speed by the drive system 12) followed by a take-offsequence (in which the autonomous vehicle 1 takes off).

During movement (i.e., flight in the example) of the autonomous vehicle1, the navigation program (e.g., operating on the navigation computer ofthe navigation system 16) provides flight control instructions to theflight control program (which is operating on the flight controlcomputer of the flight control system 15).

As will be appreciated, the navigation program has defined a flight pathwhich is based on a model (i.e., a simplified representation) of theobject 5 and the start location of the autonomous vehicle 1 with respectto the object 5 and the transition and inspection location(s).

However, the start location of the autonomous vehicle 1 with respect tothe object 5 may well be an approximation and, even if relativelyaccurate, may still be insufficiently accurate for an inspection tooccur.

Therefore, the navigation program is configured to re-assess thelocation of the object 5 with respect to the autonomous vehicle 1 duringthe flight of the autonomous vehicle 1—i.e., as the vehicle 1 performsits inspection. In other words, the location of the object 5 withrespect to the autonomous vehicle 1 is refined during operation of theautonomous vehicle 1.

The re-assessment and refinement of the location of the object 5 withrespect to the autonomous vehicle 1 may be such that the flight pathmust be altered. As will be understood, a basic alteration would be toshift the flight path in space as the relative locations are refined.However, this alone may be insufficient to ensure the safe operation ofthe autonomous vehicle 1 and that the inspection task is successfullycompleted.

For example, the refinement of the location of the object 5 with respectto the autonomous vehicle 1 may include the location of part of thatobject 5 with respect to the autonomous vehicle 1 and that part may bemoveable with respect to another part. In the case of a wind turbine,the rotational position of the blades of the turbine with respect to thetower (about an axis of the tower) and/or with respect to the nacelle(about an axis of the nacelle) may be refined such that the actuallocation is determined to be different compared to the initialinformation (e.g., initial setup parameters) provided by the initialsetup program, for example.

Refinements of this type may mean that the flight path, if followed,would now cause the autonomous vehicle 1 to impact a part of the object5 and/or may mean that the inspection system 19 does not inspect thecorrect part of the object (e.g., the camera 191 a may take a picture ofthin air rather than a wind turbine blade).

Accordingly, the navigation program is configured to redefine the flightplan (to form a re-defined flight plan) during operation of theautonomous vehicle 1—as new information about the relative position ofthe object 5 and autonomous vehicle 1 is determined.

In addition to redefining of the flight plan due to the approximateinitial relative location information provided, such flight planstypically assume ideal operating conditions—in which the autonomousvehicle 1 can and does follow the flight path set out in the plan.However, in practice, the autonomous vehicle 1 is exposed to externalforces which are unpredictable and for which compensation is difficultto provide. In the case of an aircraft, this may include the effect ofthe wind on the autonomous vehicle 1 (but similar issues arise withwatercraft due to currents and with ground vehicles due to unstable orslippery surfaces, for example).

If, when exposed to such forces and moved from the desired flight path,the autonomous vehicle 1 attempts to return to the originally desiredflight path then this may no longer be an ideal flight path and theremay be a better flight path as a result of the unintended movement.Again, identifying new ideal flight path requires the navigation programto redefine the flight plan during operation of the autonomous vehicle1.

As will be appreciated, redefining the flight plan during operation ofthe autonomous vehicle 1 is a complex task, which must be performedquickly and accurately to enable safe and efficient operation of theautonomous vehicle 1.

There is a need, therefore, to provide better flight planningtechniques.

The flight planning implemented by embodiments is based on a model ofthe object 5 and this model may be refined throughout the operation ofthe vehicle 1.

An overview of the model refinement which may be implemented byembodiments can be seen in FIG. 12 . This overview is one example and itshould be noted (as explained herein that more or few sensors may beused and more or fewer models may be generated). The refined model maythen be used in flight planning and each refinement of the model mayresult in re-processing of the flight plan and revisions thereto duringthe flight (i.e., during operation of the vehicle 1). This process maybe substantially continuous during operation of the vehicle 1 (e.g.,during flight).

Looking at FIG. 12 , an initial model (e.g., simplified representationof the object 5) may be determined and this may be performed by thenavigation program. This model may be determined based on the initialsetup parameters, for example. However, the model could be determinedthrough other methods.

This model is of the object 5 or at least part thereof and may includeone or more other objects (or parts thereof)—i.e., other than the object5 to be inspected. This initial model along with a position for theobject 5 relative to the autonomous vehicle 1 (e.g., the inspectionsystem 19) form the preliminary inputs on which the navigation is based.As will be appreciated, the position may be in the form of locationinformation which may form part of the initial model itself.

The navigation program is configured to receive data from the sensorsystem 17, e.g., from the one or more sensors 171. As shown in FIG. 12as an example, the data from one sensor 171 may be used independently(such as the data from “Sensor A” in on the left of the figure) or thedata from more than one sensor may be combined (such as the data from“Sensor A” and “Sensor B” on the right of the figure).

The data is analyzed by the navigation program to cluster that data intoobjects and/or partial objects represented by the data received from thesensor system 17.

As will be appreciated, this may include the combining of data frommultiple sensors 171 prior to clustering (see, for example, the rightside of FIG. 12 ).

The navigation program may then use the identified objects or partialobjects from the data retrieved from the sensor system 17 along with theinitial model to match the object or partial objects in the data fromthe sensor system 17 with the initial model or parts thereof. This mayinclude use of location information from the initial model as well aslocation information generated from the sensor data.

This enables the cross-identification of parts of the initial model withthe data retrieved from the sensor system 17.

As will be appreciated, the sensor data retrieved from one sensor 171 orset of sensors 171 combined may be more or less accurate than the sensordata from another sensor 171 or set of sensors 171 for various potentialreasons.

Therefore, the navigation program may generate one or more potentialrevised models, based on the sensor data (which may include locationinformation), the initial model (which may include locationinformation), and the cross-identification results.

The navigation program may be configured to combine two or morepotential revised models to derive a merged model which is then used toupdate the initial model (including new location information).

This process may be repeated such that the updated initial model is thenitself updated (e.g., becomes the initial model which is indicated atthe top of FIG. 12 ). This may be an iterative progress.

Along with updating of the model, the navigation program may also beconfigured to determine and update a flight plan, including a flightpath. Each time the model is updated, the flight plan must also bechecked and updated if required—in light of the new model. Again, thismay be an iterative process.

This general process performed by the navigation program is discussed inmore detail below. At least an initial flight plan may be determinedusing the initial model. Therefore, the discussion below begins with adiscussion of the model followed by how the flight plan may bedetermined. There is then a discussion about how the model may beupdated (see the mapping and localization processes).

As mentioned above, the navigation program uses an initial model of theobject 5 or parts thereof.

The model may comprise a collection of geometric shapes or objects whichprovide a simplified representation of the object 5.

In the example of a wind turbine as the other object 5 (and also inrelation to other objects 5), the model may comprise one or morecylinders. For example, the tower of the wind turbine may be representedby one cylinder, the nacelle may be represented by another, and each ofthree blades may each be represented by a respective cylinder.

In some embodiments, the or each geometric shape may be a convex solidthat may be sampled by a set of convex polygons stacked parallel alongone of its dimensions. In some embodiments, the stack may not be aparallel stack such that adjacent sides of two polygons are angled withrespect to each other—which may be particularly useful in relation tomodelling of curved objects 5 or parts thereof, for example. Each convexpolygon may form a slab with a thickness and the thickness of each slabmay (or may not) be the same as each other. In some embodiments, thesubtended angle between any three adjacent points (i.e., corners) of theor each polygon may be <180 degs.

The model may, therefore, include one or more convex shapes which mayencompass the object 5. The model may, however, use one or more othershapes to encompass the object 5 (in addition to or instead of convexshapes).

Indeed, in some embodiments, the model could include a point cloud, withthe points representing surface locations on the object 5. However, insome embodiments, the model is formed of only convex shapes. In someembodiments, the or each convex shape is a three dimensional convexshape. In some embodiments, the model may include one or more meshes oftwo dimensional shapes—such as a triangular or square or rectangularmesh of shapes forming the model.

The model may be formed by one or more shapes which are such that aninspection target may be defined on an edge of the or each shape and aninspection location identified by a projection therefrom onto theinspection target—a convex shape being one example thereof.

In some embodiments, the model may comprise a plurality of featurepoints 201,

^(c), which each represent the locations of one or more features of theobject 5 (see FIG. 15 , for example). The or each feature represented bya feature point 201 may be a feature which can be identified in one ormore images of the object 5, which may include one or more visible lightimages, for example.

In the case of a wind turbine, for example, as the object 5, the featurepoints 201 may represent, for example, one or more of a base of a towerof the wind turbine, a top of the tower of the wind turbine, a distaltip of the blade (or blades) of the wind turbine, and a proximal end ofthe blade (or blades). The proximal end of the blade or blades may be acentral location with respect to the blade or blades about which theblade or blades rotate (e.g., the hub for the blade or blades).

In some embodiments, see FIG. 14 for example, in addition to or insteadof the feature points 201, the model may comprise a set of feature lines202,

^(c), which extend between feature points 201 (each feature line 202 ofthe set extending between two feature points 201). The feature points201 may, therefore, provide endpoints for the feature lines 202 formingthe model. The feature lines 202 may collectively form a representationof the object 5 or a part thereof. This representation may, for thepurposes of explanation, form a stick-like or skeletal drawing of theobject 5 or a part thereof.

In the case of a wind turbine, the model—comprising one or more featurelines 202—may include:

-   -   a first feature line 202 a extending between a feature point 201        representing the base of the tower of the wind turbine and a        feature point 201 representing the top of the tower of the wind        turbine;    -   a second feature line 202 b extending between the feature point        201 representing top of the tower and a feature point 201        representing the hub; and    -   a plurality of third feature lines 202 c each extending from the        feature point 201 representing the hub to respective feature        points 201 representing distal blade tips.

A model of an object 5 in the form of a wind turbine may, therefore,include six feature points 201 and five feature lines 202.

As will be understood, the feature points 201 may be defined using athree dimensional coordinate system and, likewise, the feature lines 202may be defined in a three dimensional coordinate system. The featurelines 202 may be defined in a number of different ways—such as extendingin a straight line between two feature points 201 or extending from afeature point 201 in a specified direction for a specified distance.

The model, with its feature points 201 and/or feature lines 202, maytherefore provide a representation of one or more physicalcharacteristics of the object 5 and this may include a pose of theobject 5—such as the relative position of the blades of a wind turbinewith respect to the tower of the wind turbine—and/or a position of theobject 5 within a frame of reference. This model may be sufficientlygeneralized to accommodate various different versions of the object5—for example, different designs/configurations/shapes and sizes of windturbine.

The feature lines 202 may represent central parts of the object 5. Inparticular, the feature lines 202 may include one or more lines which donot represent edges of the object 5 but, rather, would lie inside theobject 5 if the object 5 were projected onto the model. In someinstances, this form of model may be advantageous if edges cannotreadily be detected (see elsewhere herein about how the models may beparameterized) and/or the model is expected to represent a number ofvariations of the object 5 (e.g., differentdesigns/configurations/shapes and sizes of wind turbine).

A model comprising one or more feature points 201 and/or feature lines202 of this type may be referred to as a skeletal model, for example.The skeletal model, therefore, be distinguished from a model comprisinga collection of geometric shapes or objects; however, both forms ofmodel provide a simplified representation of the object 5. In someembodiments, the model is an articulated model, such that changes in oneor more parameters of the model may result in non-linear and/ordiscontinuous movement of the at least part of the model. The skeletalmodel may include at least one parameter which may define aconfiguration of at least part of the object 5.

The navigation program may be configured to determine the model or maybe provided with the model (e.g., from the remote server 3). The modelrepresents or describes the object 5.

The model is a representation of the object 5 (or a part thereof) andmay have a nominal form dependent on the type of object 5 (e.g., a windturbine may be represented in the model by shapes for the tower,nacelle, and each blade).

The model may be provided to the navigation program as anunparameterized nominal model—i.e., a relatively generic representationof the object 5. The model may, therefore, include one or moreconfigurable parameters.

These one or more configurable parameters may place the nominal modelrelative to the autonomous vehicle 1 (e.g., to the inspection system 19)and/or the position from which the autonomous vehicle 1 will belaunched. The one or more configurable parameters may include a heightassociated with at least part of the object 5—such as a height of thenacelle of a wind turbine—relative to the autonomous vehicle 1 (e.g., tothe inspection system 19) and/or the position from which the autonomousvehicle 1 will be launched. The one or more configurable parameters mayinclude at least one relative position of a part of the object 5 withrespect to another part—such as the location of a blade of the windturbine (this may be a position relative to the tower for example). Theat least one relative position may include the heading of the windturbine and this may be relative to, for example, North. The at leastone relative position may include the position of the nacelle or bladecenter relative to the tower—alternatively, this may be a separateparameter relative to the autonomous vehicle 1 (e.g., to the inspectionsystem 19) and/or the position from which the autonomous vehicle 1 willbe launched. The one or more configurable parameters may, therefore,include the positions of one or more feature points 201 and/or one ormore feature lines 202 (their endpoints and/or lengths and/or directionsof extension)—noting the discussion herein regarding how the featurelines 202 may each be defined.

The model may be parameterized (i.e., completed by the determining theparameters associated with the model—in other words, determining thevalues of the parameters) by the initial setup program in the firstinstance, in some embodiments. Alternatively, the initial setup programmay provide information regarding the object 5 which is then used by thenavigation program to parameterize the model.

The parameterization of the model may, therefore, include the definingof one or more of the feature points 201 and/or feature lines 202 usingthe information obtained through the initial setup program. In someembodiments, the parameterization of the model may include theprocessing of one or more images of the object 5. The or each image mayinclude one or more visible light images, for example.

The or each image may be captured by a sensor 171 of the sensor system17 (which sensor 271 may include a camera). In some embodiments, the oreach image may be captured by an inspection sensor 191 of the inspectionsystem 19 (which inspection sensor 191 may include a camera 191 a).Indeed, in some embodiments, a camera 191 a may be shared between thesensor system 17 and inspection system 19 such that a single camera mayprovide images for both systems. In some embodiments, the or each imagemay be provided independently of the autonomous vehicle 1. For example,the or each image may be captured by a camera which does not form partof the autonomous vehicle 1 but which may form part of the base station2, for example, or which may be captured by a handheld camera operatedby a human user.

The or each image may include an image of at least part of the object 5and which may include substantially all of the object 5.

In some embodiments, such as those using a skeletal model, the model maybe parameterized using one or more such images (and may also useinformation from the initial setup program) using the mapping and/orlocalization processes described herein in relation to skeletal models.

In order to determine an initial flight plan, the navigation program maytake into account the operational characteristics of the inspectionsystem 19.

The navigation program may be configured to receive, or may bepre-programmed with, information regarding the operationalcharacteristics of the inspection system 19 (e.g., the or eachinspection sensor 191). In some embodiments, the inspection system 19 isconfigured to provide the navigation program with this information(e.g., the inspection system computer 192). In some embodiments, thenavigation program may obtain the information from the remote server 3or may already have the information as part of the navigation program.

The operational characteristics of the inspection system 19 may include,for example, one or more of:

-   -   a maximum distance the inspection system 19 can be from the        object 5 for an accurate inspection (e.g., at a desired minimum        resolution),    -   a minimum distance the inspection system 19 can be from the        object 5 for an accurate inspection,    -   a field of view of the inspection system 19 (which may be linked        to a distance between the inspection system 19 and the object        5),    -   a resolution of the inspection system 19,    -   a variation in resolution of the inspection system 19 with        respect to an angle between the inspection system 19 and the        inspection target taking into account any curvature of the        object 5 at the inspection target (which may influence,        therefore, an effective field of view of the inspection system        19),    -   a minimum setup time for an inspection operation (i.e., a        minimum time at which the inspection system 19 has to be at the        inspection location before an inspection operation can occur),    -   a minimum inspection operation time (i.e., a minimum time at        which the inspection system 19 has to be at the inspection        location for an inspection operation to occur),    -   a range of movements (e.g., changes in orientation) of the field        of view (or effective field of view) of the inspection system 19        with respect to the vehicle 1 (in some embodiments, the or each        inspection sensor 191 may have constrained changes in        orientation with respect to the vehicle 1), and    -   a minimum unset time (i.e., a minimum amount of time for the        inspection system 19 to adopt a mode of operation suitable for        movement of the autonomous vehicle 1—which may include covering        an inspection sensor 191 for example (e.g., to avoid        contamination during the flight) and this may only apply to some        inspections).

The field of view (or effective field of view) of the inspection system19 may include an area or volume which can be covered by the inspectionsystem 19 from one inspection location (e.g., at a particular distancefrom the inspection system 19) and may include a depth range. Forexample, an inspection sensor 191 may have a field of view (or effectivefield of view) associated with a particular distance of the inspectionsensor 191 from the object 5 but that field of view may encompassvariations in that distance (i.e., depth) within a predetermined range.One example would be a camera 191 a, which can be focused to aparticular distance but parts of the object 5 which are at that distanceor within a given range of variation of that distance (i.e., depthvariation) from the camera 191 a will also be in sufficient focus forthe inspection operation.

These operational characteristics may be provided for the inspectionsystem 19 as a whole or in relation to the or each respective inspectionsensor 191.

In order to determine an initial flight plan, the navigation program maytake into account the operational characteristics of the vehicle 1 (inaddition to or instead of the operational characteristics of theinspection system 19).

The navigation program may be configured to receive, or may bepre-programmed with, information regarding the operationalcharacteristics of the vehicle 1. In some embodiments, the fight controlsystem 15 is configured to provide the navigation program with thisinformation. In some embodiments, the navigation program may obtain theinformation from the remote server 3 or may already have the informationas part of the navigation program.

The operational characteristics of the vehicle 1 may include, forexample, one or more of:

-   -   possible degrees of freedom of movement of the vehicle 1;    -   one or more costs associated with one or more movements of the        vehicle 1;    -   a maximum altitude of the vehicle 1 (or other maximum range of        the vehicle 1); and    -   a power capacity of the vehicle 1.

The navigation program may be configured to determine one or moreinspection targets relative to the model (i.e., representation) of theobject 5 using one or more of the operational characteristics of theinspection system 19—see FIG. 6 , for example, which shows a collectionof inspection targets—and/or one or more operational characteristics ofthe vehicle 1.

An inspection target may be a point which is on a surface of the modelwhich is to be the target of an inspection operation by the inspectionsystem 19 of the autonomous vehicle 1.

The or each inspection target may be based at least in part on, forexample, one or more of: the initial setup parameters, an aim to inspectsubstantially an entire surface of the object 5 or a part of the object5 (such as the blades of a wind turbine), operator defined inspectiontargets (e.g. identified by the operator using the operator interface 42which may display a graphical representation of the model), and one ormore previous inspections of the object 5 (such information may beprovided by the remote server 3, for example).

The or each inspection target may be determined using the one or moreoperational characteristics of the inspection system 19 so as, forexample, to ensure the desired coverage of the inspection (e.g., byvirtue of the combination of inspection operations (although there mayonly be one in some embodiments)). Accordingly, the or each inspectiontarget may be selected to ensure that one or more of the inspectionoperations have overlapping coverage at one edge, at least. In someembodiments, a desired non-overlapping spacing of inspection targets(i.e., negative overlap or negative distance/spacing) may be requiredfor the inspection operation. This may be the case, for example, inrelation to inspection operations comprising spot inspections.

For example, the navigation program may be configured to use the fieldof view (or effective field of view) characteristic to define acollection of inspection targets such that a predefined portion (whichmay be all) of the object 5 is covered by the inspection operations withoverlaps between the inspection data to allow the inspection data fromone inspection operation to be combined with the inspection data ofanother inspection operation. This may, of course, entail capturing (asthe inspection data) a series of images (using the camera 191 a) suchthat the images can be stitched together to form a larger image. Thesame technique (i.e., data stitching) may be used with other forms ofinspection data.

In some embodiments, the navigation program may be configured to defineinspection operations which have a degree of overlap which is dependent,at least in part, on an angle and/or distance of the inspection system19 (e.g., an inspection sensor 191) with respect to the object 5 withinthe area of overlap. As will be appreciated, the resolution of theinspection operation 19 may vary depending on this information and sothere may be a requirement for greater overlap between inspectionoperations if each operation has a relatively low resolution in the areaof the overlap. The inspection data for overlapping inspectionoperations may be combined to improve the effective resolution of theinspection data in the overlap area.

The navigation program may be configured to determine one or moreinspection locations and may be configured to determine at least onetransition location based on the inspection targets and the model (i.e.,representation) of the object 5. In other words, the navigation programis configured to determine the flight plan.

The navigation program may be configured to determine multiple differentflight plans and may seek to determine an optimal flight plan based onone or more cost functions/operations.

The navigation program may be configured to determine the or each flightplan based at least in part on one or more operational constraints. Theor each operational constraint may be predetermined, may be part of theinitial setup parameters, may be obtained from the remote server 3, ormay be input by an operator (e.g., using the interface 42).

A first example of an operational constraint is a range constraint. Therange constraint may determine how close the autonomous vehicle 1 canbe, during flight (in accordance with the flight plan) to the object 5.This range constraint, therefore, effectively forms an invisible wallbetween the autonomous vehicle 1 and the object 5 which is intended toreduce or substantially eliminate the risk of the autonomous vehicle 1impacting the object 5.

The range constraint could be a minimum distance (or range) from anypart of the model of the object 5. As such, the autonomous vehicle 1 maymove around the object 5 so long as this minimum range is maintained.This may be referred to as a constant range constraint.

The range constraint could be defined as a plane relative to the object5 (in other words a virtual plane which is a predetermined distance fromthe closest part of the model to the autonomous vehicle 1 through whichno part of the object 5 extends (and, in some embodiments, from whichthere is no part of the object 5 which is within a minimum distance)).This may be referred to as a planar range constraint.

In some embodiments, the range constraint is defined as a predeterminedrange from an inspection target, a plurality of the inspection targets,or all of the inspection targets. This range may define, therefore, aspherical locus to which the vehicle 1 is constrained or is to beconstrained. This spherical locus may be within a predetermined range ofanother inspection target and a range constraint may, therefore, bedetermined by an amalgamation of a plurality of loci. As will beunderstood, this locus may intersect the object 5 itself in one or morelocations and the navigation program may be configured—using theprocesses described herein—to discount inspection locations from suchparts of the locus.

As will be appreciated, the range constraint may be selected based atleast in part on the operational characteristics of the inspectionsystem 19 (at least such that the inspection system 19 is able toperform the inspection operation in relation to the object 5 at theminimum distance to the object 5 determined by the range constraint (orcloser)).

In some embodiments, two range constraints may be applied—one on eitherof two opposing sides of the object 5, for example. As such thenavigation program may be configured to define a flight path which movesthe autonomous vehicle 1 past at least part of the object 5 from onerange constraint to another range constraint. These two rangeconstraints may both be planar range constraints. These two rangeconstraints may effectively provide a first range constraint associatedwith a first side of an object 5 and a second range constraintassociated with a second side of an object 5, wherein the two sides maybe opposing sides (e.g., front and back sides of the object 5).

As will be appreciated, in some embodiments, the navigation program mayseek to maintain a relatively close proximity to the range determined bythe range constraint—to seek to ensure an effective inspectionoperation. Accordingly, which form of range constraint is used mayinfluence the inspection locations relative to the inspection targets.In some embodiments, the or each range constraint may be selected basedon one or more aspects of the inspection operation and/or theoperational characteristics of the inspection system 19—e.g., to ensurethe inspection operation completes successfully.

In some embodiments, the or each range constraint defines an operationalzone in which the vehicle 1 is permitted to operate or in which thenavigation program is permitted to define the flight path. In someembodiments, this zone is a plane (which may be a curved plane) or locus(e.g., to which the flight path is constrained or is to be constrainedby the navigation program).

As will be understood, the predetermined geometrical relationship may bedetermined using a distance from the inspection location to theinspection target and an angle of incidence (of an inspection axis ofthe inspection system 19) on the curved surface from the inspectionlocation. These two factors must be balanced against each other by thenavigation program. In particular, to improve inspection resolutionusing a different angle of incidence typically requires the inspectionsystem 19 to be further away from the inspection target along the rangeconstraint plane. The navigation program may seek, therefore, tooptimize the loss of resolution due to this increased distance to theinspection target with the increase in resolution due to the improvedangle of incidence. The inspection axis may be defined as central axisof the field of view of the inspection system 19 (and inspectionsensor(s) 191), extending from the inspection system 19 (and inspectionsensor(s) 191) to the inspection target.

There may be other constraints as well which may include minimum rangesto other objects (other than the object 5 being inspected) or otherparts of the object 5 being inspected.

The range of the autonomous vehicle 1 to the object 5 (and any otherobjects in some embodiments) may be determined by the sensor system17—as will become apparent.

The navigation program may also determine, as part of the generation ofthe flight plan, an orientation of the inspection system 19 and, inparticular, the one or more inspection sensors 191 with respect to theobject 5. It may be that at least one of the one or more inspectionsensors 191 is not omnidirectional and must be directed (i.e., pointedor positioned or oriented) towards the object 5 and, in particular, theinspection target. This may be the case in relation to a camera 191 a,for example.

Accordingly, the navigation program may associate the or each inspectionlocation with an inspection system orientation (as can be seen, forexample, in FIG. 7 ). In some instances, the inspection system 19 (or atleast one or more of the inspection sensors 191) may have a fixedorientation with respect to the frame 11 and, therefore, the inspectionsystem orientation may be a required orientation of the autonomousvehicle 5 with respect to the object 5 at that inspection location(e.g., to point the or each inspection sensor 191 towards the inspectiontarget). In some embodiments, at least one of the one or more inspectionsensors 191 may be configured to move with respect to the frame 11 tochange its relative orientation with respect thereto. In someembodiments, therefore, the inspection system orientation may include(in addition to or instead of an orientation of the autonomous vehicle 1with respect to the object 5) an orientation of the or each inspectionsensor 191 with respect to the object 5 and/or frame 11—and this may beprovided by the navigation program to the inspection system computer 192for use by the inspection system computer 192 to control a movement ofthe or each inspection sensor 191 relative to the object 5 and/or frame11 as the flight plan is executed.

In some embodiments, the navigation program is likewise configured touse other inspection location data (the orientation being one example)which may be used by the inspection system 19 as described herein. Thatinspection location data may be used, in short, by the inspection system19 to configure the or each inspection sensor 191 to perform itsinspection operation (and may include a distance of the object 5 fromthe inspection system 19/autonomous vehicle 1 at the inspectionlocation, for example). This enables, for example, the inspection system19 be optimized in flight in accordance with the flight plan—see belowfor more details.

The navigation program may be configured to begin to determine theflight plan as, therefore, a point cloud comprising at least oneinspection location (and, in some embodiments, at least one transitionlocation). The flight plan will likely also include a start point and anend point (which may be one and the same location, which may be referredto as a home location and which may be the location at which theoperator has placed the autonomous vehicle 1).

The navigation program may be configured to perform three operations inrelation to the flight plan:

-   -   (i) a location safety assessment;    -   (ii) an effectiveness assessment; and    -   (iii) a movement safety assessment.

In some embodiments, these three operations may be performed in relationto the or each inspection location (and may be performed in relation toone or more transition locations).

In the location safety assessment, the navigation program is configuredto determine if a particular location is a safe location for theautonomous vehicle 1 to be. This may include, for example, assessment ofthe one or more constraints discussed above. In some embodiments, thisassessment is based on one or more environmental factors (such as windspeed, wind direction, current speed, current direction, angle of slopeon which the autonomous vehicle 1 is located, etc.). In particular, asafe distance from a particular object (be it the object 5 to beinspected or another object) may be smaller in low wind speed conditions(compared to high wind speed conditions) and safe distances may varydepending on the direction of the wind.

If a location (inspection or transition) is found not to pass (i.e., tofail) the location safety assessment, then that location may be deletedor moved (i.e., are rejected) by the navigation program.

In some embodiments, the location safety assessment may include theassigning of a location safety score of a location based on one or moreconstraints and/or environmental factors—with the location safety scoreindicating the perceived safety risk of that location.

The effectiveness assessment is performed at least in relation to the oreach inspection location. In accordance with this assessment, thenavigation program is configured to determine whether the relevantinspection target is likely to be inspectable by the or each inspectionsensor 191 of the inspection system 19 from that inspection location(e.g., with a required minimum resolution and without obstruction). Inrelation to some objects 5, such as a wind turbine for example, theposition of one part of the object 5 (e.g., a blade) with respect toanother (e.g., a tower) may determine whether a part of that object 5will be between the inspection target and the inspection location (i.e.,block the inspection target). This may be a particular problem for windturbines with an inspection of a blade which is located adjacent andsubstantially parallel to the tower (thus limiting access to a portionof the blade closest to the tower).

In order to perform the effectiveness assessment, the navigation programmay be configured to project (e.g., using ray tracing) the field of viewof the or each inspection sensor 191 onto the object 5 (i.e., the modelof the object 5). In some embodiments, the navigation program may beconfigured to perform this projection onto the surface of the model ofthe object 5 which is of interest and may exclude one or more (or all)portions of the field of view of the or each inspection sensor 191 whichextend beyond this area of interest. This may help to reduce therejection of otherwise suitable inspection locations due to blocking ofan unrequired portion of the field of view of the or each inspectionsensor 191, for example. Accordingly, locations which do not pass (i.e.,fail) the effectiveness assessment are deleted or moved (i.e., arerejected) by the navigation program. In some embodiments, theeffectiveness assessment may include the assigning of an effectivenessscore based on the perceived effectiveness of the inspection operationfrom that inspection location.

The movement safety assessment is performed by the navigation program todetermine whether a location (which may be an inspection location butmay be a transition location in some embodiments) is safe to move to andaway from. This may include assessment of the constraints mentionedabove. Again, in some embodiments, the movement safety assessment mayinclude the assigning of a movement safety score based on a perceivedsafety risk associated with the movement. One or more transitionlocations may be added to the flight plan as part of the movement safetyassessment to ensure that the movement is safe for a given location.

In order to aid in one or more of these three operations, the navigationprogram may use one or more additional models of the object 5 which maytake different forms.

The navigation program may, therefore, be configured to generate a pointcloud which includes a plurality of inspection locations and which mayinclude one or more transition locations. The navigation program has notyet, at this stage of some embodiments, sequenced the locations into aflight plan.

As will be understood, from the point cloud of locations (which may beonly inspection locations but might include one or more transitionlocations), a large number of possible flight plans could be determined.Typically, each such flight plan would include additional transitionallocations, for example, to define the flight path between the inspectionlocations.

The point cloud is assessed by the navigation program to select one ofthe possible multitude of possible flight plans. This process may bereferred to as the flight plan selection process.

The flight plan selection process may use one or more of a number ofdifferent methods.

The flight plan selection process may include a search for differentpossible flight paths. This may be performed using a suitable searchapproach such as a breadth first search, a depth first search, a greedyalgorithm, Dijkstra's algorithm, an A* algorithm, and/or any suitablegraph search technique. In some embodiments, the flight plan selectionprocess includes an operator selection of one or more portions of theflight path (which may be input using the operator interface device 4).

The flight paths which are determined by the navigation program may,therefore, include one or more transition locations which are determinedby the flight plan selection process, and one or more inspectionlocations (and may include one or more predetermined transitionlocations).

The different possible flight paths determined by the navigation programusing a search approach are then costed based on a cost criterion andordered according to that cost criterion.

The costing process of the navigation program may be based on one ormore possible cost criterion such as distance, time, power consumption(from the power source 14), and the like. In some embodiments, thecosting process may order the different possible flight paths based ontime (i.e., flight or travel time).

In some embodiments, one or more cost criteria are converted into costsin common different criterion such as power consumption or distance. Insome embodiments, for example, one or more non-time related costcriteria are converted into time-based costs for assessment in thecosting process. In some embodiments, a distance of the autonomousvehicle 1 to the object 5 may be included in the costing process(including, for example, a predicted speed of the autonomous vehicle 1).In some embodiments, an angular deviation between an angular position ofthe autonomous vehicle 1 and the angular position of one or more of theinspection sensors 191 may be included in the costing process(including, for example, an angular speed of the autonomous vehicle 1 ormounting device of the inspection system 19). In some embodiments, atime penalty may be applied for transitions between movement to inspectone part of the object 5 with respect to another (e.g., the blades withrespect to the tower, and/or a front of the object 5 with respect to aback of the object 5).

The costing process may be configured to select the lowest cost flightpath from the different flight paths.

The costing process may take into account the location safety scoreand/or the effectiveness score and/or the movement safety score—ifdetermined (see above).

The costing process may take into account (particularly for a planarrange constraint) the balance between the distance from the inspectiontarget and the angle of incidence—as described above. As will beappreciated, a greater distance from the inspection target may provide awider effective field of view of the inspection system 19, which maymean fewer inspection operations are required but that each inspectionoperation may include inspection data of a lower resolution in relationto the inspection target. Therefore, more inspection operations may beneeded to provide the required overlap to improve the effectiveresolution of the inspection data. The navigation program may beconfigured to use these factors in the costing process.

In some embodiments the navigation program may be further configured toperform a selected flight path safety process. In accordance with thissafety process, the navigation program may be configured to confirm theflight path meets one or more safety rules. The or each safety rule mayinclude, for example, a minimum range from the object 5 and/or anotherobject (other than the object 5). The safety process may includeanalysis of the or each transition location (which may be only thosetransition points determined by the flight plan selection process, forexample).

In accordance with this safety process, the navigation program may useray tracing techniques to consider the path between sequential locationsin the flight path (be they between two transition locations, onetransition location and an inspection location, or between twoinspection locations, for example). The navigation program may beconfigured to determine if that flight path (as represented in the raytracing technique) would intersect the object 5 (or another object) orwould pass within the minimum range of the object 5 (or other object) orwould pass into a restricted geofence or out of a permitted geofence. Ifthis in the case, then the navigation program may be configured to moveone or more transition locations (determined in the flight plan sectionprocess, for example), to delete one or more such transition locations,and/or to add one or more additional transition locations—all with aview to reducing the risk of the autonomous vehicle 1 impacting theobject 5 (or other object). Any moved transition locations, for example,or otherwise altered flight path may be re-assessed by the navigationprogram using the assessments mentioned above. In addition, oralternatively, any new flight path may be compared, again, against otherflight plans (e.g., using the costing described herein) to determine ifit is still to be the selected flight path or if a different path (and,therefore, plan) should be adopted.

The navigation program may be configured to record and store informationabout the operation of the vehicle 1 during implementation of a flightplan—i.e., as the vehicle 1 moves along the flight path performinginspection operations. This information may be stored as a log, forexample, and may be sent to the communication system 18 for transmissionand storage remotely from the vehicle 1, for example. This remotestorage may be in the base station 2 and/or remote server 3, forexample.

In some embodiments, the navigation program may be configured to resumean interrupted flight plan. For example, the navigation program may beconfigured (e.g., under operator instruction using the operatorinterface device 4) to select information stored about a partiallycompleted flight plan. The navigation program may be configured toresume this partially completed flight plan and may seek to avoidre-inspection of any inspection locations already the subject of aninspection operation (during the partial completion of the flight plan).

This may enable, for example, a flight plan which is terminated orsuspended mid-flight to be completed efficiently. A mid-flighttermination or suspension may be due, for example, to power or timeconstraints or a safety issue being identified.

In some embodiments, the navigation program may be configured to resumethe flight plan without recalculation of the flight plan—other than thenew portion of the flight plan required to move the vehicle 1 to whatwill be the starting location of the resumed flight plan.

In other embodiments, the navigation program may be configured torecalculate the flight plan for the inspection targets which were notthe subject of inspection operations during the terminated or suspendedflight plan. As will be appreciated, the navigation program may beconfigured to determine new inspection locations and/or a new flightplan as a result and this may take into account a potential change inthe object 5—such as movement of one part thereof with respect toanother part. Changes in the object may be determined by the navigationprogram as part of updating the model as discussed herein.

In other words, the navigation program may be configured to generate anew flight plan based at least in part on the stored information about aprevious partially implemented flight plan (i.e., a flight plan which avehicle 1 began to follow but which the vehicle 1 did not complete).

The navigation program may be configured, as described above, (via thesafety process and/or the costing process) to determine and select aflight plan. This flight plan may be, however, based initially onrelatively rudimentary information about the location of the autonomousvehicle 1 relative to the object 5 and the configuration of the object5.

In other words, this flight plan is initially based on a model (i.e., asimplified representation) of the object 5 and approximate relativelocation information for the object 5 and the autonomous vehicle 1.

Therefore, the navigation program may be further configured to receiveinformation from the sensor system 17 from which one or more of thegeographic location of the autonomous vehicle 1, the geographic locationof the object 5, the relative location of the autonomous vehicle 1 tothe object 5, and the configuration of the object 5, may be updated orrefined. Using this information, the model of the object 5 used for theflight planning may be altered and this may, in turn, cause thenavigation program to revise the flight plan using the processesdescribed herein to regenerate the flight plan or a part thereof.

Accordingly, the navigation program may be configured to perform amapping process and/or a localization process. The mapping and/orlocalization processes may be performed during flight (i.e., movement)of the autonomous vehicle 1. As discussed, this may result in a newmodel—which may be the same nominal model as the initial model butparameterized with one or more updated parameters.

The mapping process may include one or more of determining or refining aconfiguration of the object 5. The localization process may includedetermining a relative location of the autonomous vehicle 1 with respectto the object 5—which may include determining a location of a model ofthe object 5 determined in the mapping process (which may be a locationrelative to the vehicle 1 (e.g., relative to the inspection system 19)and may take into account the orientation of the vehicle 1 (e.g., yaw,pitch, and roll)). These processes may be executed during movement ofthe autonomous vehicle 1 in parallel or in a repeating sequentialmanner.

In some embodiments, the mapping and/or localization processes mayinclude the steps of receiving sensor data from the sensor system 17(e.g., from one or more of the sensors 171). This sensor data mayinclude location information and information from, for example, aranging sensor 171 or camera 171 of the sensor system 17. As mentionedherein, the camera 171 of the sensor system 17 may also be the camera191 a of the inspection system 19, for example.

The mapping and/or localization processes may include the processing ofthe received sensor data to cluster or split the data into two or moreclusters or portions—i.e., to form clustered sensor data.

The clustered sensor data may then be analyzed as part of the mappingand/or localization process to identify one or more parts of the object5 represented by each of the clusters or portions in the clusteredsensor data.

The mapping process may include the determining of the relative position(which may include a location and/or orientation or other configuration)of one cluster or portion with respect to another. This relativeposition represents the relative position of the correspondingidentified part or parts of the object 5 with respect to each other.Accordingly, the mapping process may determine information about theconfiguration of the object 5. This may include determining one or moretransforms to shift the initial model to match the information about theconfiguration of the object 5. This may include adjustments to one ormore of the parameters of the initial model.

The localization process may include the determining of the relativeposition (which may include a location and/or orientation or otherconfiguration) of one cluster or portion with respect to the vehicle 1(e.g., with respect to the inspection system 19). Accordingly, thisrelative position represents the relative position of the object 5 (or apart thereof) with respect to the vehicle 1 (e.g., with respect to theinspection system 19). The relative position may be determined in termsof a global frame of reference and may be referred to as the objectlocation, for example. This may include determining one or moretransforms to shift the initial model to match the information about therelative position of the object 5. This may include adjustments to oneor more of the parameters of the initial model.

In some embodiments, the sensor data may include one or more images,which may be visible light images or infrared or ultraviolet lightimages. The mapping and/or localization processes may be performed onthis sensor data in a similar manner, with parts of the imagesidentified as corresponding to parts of the initial model. Thisinformation may then be used as described above in the determining stepsof the mapping and/or localization processes. This may result in anotherin-flight model which may be combined with other models and/or used tomodify the initial model for further use by the navigation program.

For example, in the case of the use of a skeletal model, a localizationprocess (which may be part of the operation of the navigation program)may, in some embodiments, include the receipt of one or more imagesincluding image data relating to at least part of the object 5 (and, insome instances, an image may include image data relating tosubstantially all of the object 5).

The localization process may take the or each image and pass the imagedata from the or each image through a convolutional neural network (CNN)300 (see FIG. 16 , for example, which shows a CNN architecture alongwith input and output images) to process the image data into a formwhich can be matched to the skeletal model so that the location of theautonomous vehicle 1 can be determined.

The CNN 300 may, therefore, take an image (which may be a multichannelimage) as an input and produce another image (which may be amultichannel image) as an output. The input image will be referred to asinput image data and the output as output image data. Both the input andoutput image data represent the object 5 (or a part thereof) and may bein the form of a skeletal and/or point-based representation of theobject.

The input image data may be in the form of a red-green-blue image data.In other embodiments, the input image data may be of a different form.

The CNN 300 may use a convolutional-deconvolutional architecture togenerate the output image data from the input image data.

Accordingly, in some embodiments, the input image data is successivelyconvolved and downsampled (e.g., using a pooling technique such asmax-pooling up to a bottleneck of CNN 300). These processes are thenfollowed, in some embodiments, by convolution and upsampling (e.g., upto its original spatial dimensions).

As such, the CNN 300 may be of an architecture which may include anencoder 301 (in the form of a sequential series of convolutional layers301 a, each followed by a pooling layer 301 b) and a decoder 302 (in theform of a sequential series of linear upscaling layers 302 a, eachfollowed by a convolution layer 302 b).

In some embodiments, this type of architecture is beneficial in thatvisual information (in the input image data) from across the spatialrange of the image input data is brought together in a deep part of thenetwork to provide feature rich information to an output of the decoder302 (which is a deconvolutional part of theconvolutional-deconvolutional CNN 300 architecture).

The feature lines 202 and/or feature points 201 may be classified in themodel based on the part of the object 5 which the feature lines 202and/or feature points 201 represent. In the case of a wind turbine asthe object 5, the first feature line 202 a may be a first class, thesecond feature line 202 b may be a second class and the plurality ofthird feature lines 202 c may collectively be a third class. In someembodiments, however, each of the third feature lines 202 c has its ownclass. In some embodiments, the feature points 201 may be classifiedwith each feature point 201 forming a class. In some embodiments,however, the feature points representing each distal blade tip may becollectively one class.

The output image data produced by the CNN may be in the form of aprediction of the projection of the skeleton model, given the pose andparameters of the camera from which the input image data was captured.Each channel of the output image data may correspond to a differentclass of the skeleton model. In the some embodiments, the output imagedata may, therefore, include seven channels which correspond to theseven classes in the skeleton model of some embodiments.

In some embodiments, the input image data is captured by a camera and isassociated with metadata. This associated metadata may include locationand/or direction data. In some instances, the input image data has beencaptured by a camera which is carried by the autonomous vehicle 1 (e.g.,by the camera 191 a or one of the one or more sensors 171). As such, themetadata may be determined by one of the one or more sensors 171 and mayinclude, for example, location and/or direction data obtained using theradio navigation sensor. The location and/or direction data may includeinformation representative of a pose of the camera which captured theinput image data.

The metadata may be used to assist the CNN in its operation. This may beachieved by using the metadata and the skeletal model (i.e., to form aprior). In particular, in some embodiments, the localization process mayinclude generating an estimate of the skeletal model projected onto theinput image data or otherwise into the frame of reference to the cameraor other sensor which captured the input image data (i.e., an imagingsensor).

The localization process may be configured to use the metadata and theparameterized skeletal model (which may be received as part of thelocalization process), along with one or more parameters associated withthe imaging sensor (e.g. inspection system parameter) which generatedthe input image data, in order to determine a projection of the skeletalmodel into the frame of reference of the imaging sensor or otherwiseinto the input image data. The one or more parameters may include, forexample, the intrinsic parameters of the camera (or other imagingsensor). This projection of the skeletal model will generally bereferred to as the projected skeletal model.

The intrinsic parameters of the camera (or other imaging sensor) mayrefer to, for example, nine values which determine the camera (or otherimaging sensor) intrinsic matrix and are used for the projection of 3Dpoints into a 2D image plane. These parameters are typically a fixedproperty of the camera (or other imaging sensor) and are typicallypredetermined parameters. In some embodiments, a set of distortionparameters may be used. This set of distortion parameters may be used inconjunction with the intrinsic parameters of the camera or other imagingsensor when mentioned herein (e.g., to project the 3D points into the 2Dimage plane). The set of distortion parameters may include a pluralityof parameters which are intended to correct the input image data inlight of distortions in a lens of the camera (or other imaging sensor).In some embodiments, the plurality of parameters forming the set ofdistortion parameters may include 4, 5, 8, 12, or 14 parameters. The setof distortion parameters may, similar to the intrinsic matrix of thecamera (or other imaging sensor), be a predetermined property of thecamera (or other imaging sensor). The set of distortion parameters may,therefore, be predetermined parameters.

In some embodiments, the localization process is configured to perform asmoothing operation (such as Gaussian smoothing) on the projectedskeletal model.

The projected skeletal model may then be appended to the input imagedata as additional channels of the multi-channel input image data, insome embodiments. The first feature line 202 a projection may form onechannel, the second feature line 202 b projection may form a secondchannel, and the plurality of third feature lines 202 c as projected mayform a third channel. In the case of embodiments with red-green-blueinput image data, this will form three channels (e.g., fourth-sixthchannels). The feature point 201 projections may each form a channel(again with all the projections of the distal blade tips forming onechannel).

Therefore, additional channels may be added by the localization processto the input image data prior to processing of that input image data bythe CNN 300. This may, due to the accuracy of the projection of theparameterized skeletal model, aid in the operation of the CNN 300.

Training of the CNN 300 may have been implemented prior to thelocalization process described above in relation to the skeletal model.Training may be based on real data or simulated data. Training may beundertaken using an Adam optimizer but other optimizers are alsoenvisaged.

The localization process can, therefore, generate an output image datafrom which the pose of the autonomous vehicle 1 can be determined and/orrefined. In order to do this, a pose graph is used by the localizationprocess. This process may be a pose optimization process.

A pose graph may be used to estimate the position and orientation (i.e.,pose) of the imaging sensor of the autonomous vehicle 1 (relative to theobject 5 and/or in absolute terms) and is an example of pose data.References herein to a pose graph may, therefore, refer to pose data ingeneral. Accordingly, as the autonomous vehicle 1 performs an inspectionof the object 5 a new node or keyframe may be added to a pose graph.These may be added at intervals during the operation of the vehicle 1.In some embodiments, these intervals may be predetermined regularintervals (i.e., with a constant interval timing). In some embodiments,the interval timing may be dynamically determined based on one or moreoperational aspects of the autonomous vehicle 1. These one or moreoperational aspects may include distance travelled by the vehicle 1(which may be a rotational and/or linear distance travelled), and/oroverlap between image data gathered by the imaging sensor (e.g., thecamera frustum overlap).

As will be appreciated, the orientation of the imaging sensor withrespect to the frame 11 of the autonomous vehicle 1 and/or with respectto any sensor 171 which is used to determine the location of theautonomous vehicle 1 (such as radio navigation sensor) may be varied insome embodiments—e.g., if the imaging sensor is mounted using a moveablemounting. The relative orientation of the imaging sensor with respect tothe frame 11 and/or the sensor 171 used to determine the location of theautonomous vehicle 1 may be known from a sensor associated with themounting for the imaging sensor. Accordingly, this information may beused to determine the pose of the imagining sensor when provided with apose of the vehicle 1. In some embodiments, the imaging sensor is fixedwith respect to the frame 11 and/or any sensor 171 which is used todetermine the location of the autonomous vehicle 1; therefore, the poseof the vehicle 1 is also the pose of the imaging sensor.

Each node added to the pose graph may contain an estimate of the pose ofthe camera or other sensor of the autonomous vehicle 1 and this estimatemay be an absolute pose estimate. The pose graph, therefore, mayrepresent a plurality of poses.

Each pose may be defined as:T _(i)=(R _(i) ,t _(i))  (1)

-   -   where R_(i) is the orientation of the autonomous vehicle 1, and    -   t_(i) is the position of the vehicle 1 in three dimensions.

As mentioned, each pose may be an absolute pose and may be determined bythe navigation program, for example. The absolute pose may be determinedusing one or more of the sensors 171 of the sensor system 17. This mayinclude, for example, the use of the radio navigation sensor. Therefore,this pose may be referred to as a measured pose of the imaging sensor ofthe autonomous vehicle 1.

The poses forming the pose graph may be optimized under a set ofconstraints with a view to converging the poses to the true location andorientation, T_(i)*, of the imaging sensor of the autonomous vehicle 1.

The set of constraints (which may be referred to as graph constraints),may be formed based at least in part on estimates of the pose of theautonomous vehicle 1 (and/or the imaging sensor of the autonomousvehicle 1) and an output of the CNN (i.e., the output image data).

The estimates of the pose of the imaging sensor of the autonomousvehicle 1 at each node (e.g., the measured pose) may not be used in theoptimization process. Instead, in accordance with some embodiments,consecutive poses (or substantially consecutive poses (i.e., notimmediately consecutive poses)) are paired to determine a relative pose.For example, relative pose offset is determined by the poses atkeyframes T_(i) and keyframe T_(i−1):

$\begin{matrix}{T_{i,{i - 1}} = {\begin{bmatrix}t_{i,{i - 1}} \\q_{i,{i - 1}}\end{bmatrix} = \begin{bmatrix}{R_{i}^{T}\left( {t_{i - 1} - t_{i}} \right)} \\{q_{i}^{- 1}*q_{i - 1}}\end{bmatrix}}} & (2)\end{matrix}$

-   -   where q_(i) is a quaternion representation of the orientation        R_(i) of the vehicle 1.

The use of relative pose changes may be beneficial as, for example, overa short period of time errors in the measured pose are likely to belower than over a longer period of time.

Optimization of the pose graph may be undertaken using a cost function,E, which is configured to determine a residual error between an expectedpose, {circumflex over (T)}_(i), given the current state of theoptimized pose graph and the measured pose of the imaging sensor of theautonomous vehicle 1:

$\begin{matrix}{E = {{\sum\limits_{i}\; e_{i,{i - 1}}^{T}} + {\sum\limits_{i}\; e_{i}^{I}}}} & (3)\end{matrix}$

The optimal set of poses are those which minimize this function. Theoptimized set of poses forming a new state of the optimized pose graph(effectively replacing the former state of the pose graph).

In this cost function, E, e_(i,i−1) ^(T) is intended to compare therelative pose of the expected poses, {circumflex over (T)}_(i,i−1), withcorresponding ones of the measured poses:

$\begin{matrix}{e_{i,{i - 1}}^{T} = {C\begin{bmatrix}{{\hat{t}}_{i,{i - 1}} - t_{i,{i - 1}}} \\{2 \times {{Vec}\left( {{\hat{q}}_{i,{i - 1}}*q_{i,{i - 1}}^{- 1}} \right)}}\end{bmatrix}}} & (4)\end{matrix}$

-   -   where Vec corresponds to the vector part of the quaternion        rotation, and C is a diagonal matrix weighting the different        elements of the cost.

The weighting of the different elements of the cost, C, may bedetermined using, for example, covariances of the measured poses, or maybe set based on expected errors in the measured poses. Each term may beweighted in translation error by β^(t) and those in rotation error byβ^(q).

As a result, e_(i,i−1) ^(T) may be a 6D vector containing the residualsin rotation and translation.

A cost function for the image measurements may be based onpoint-to-point correspondences which are established differentlydepending on the types of image measurements and, in particular, whetherthe skeletal model (or part being considered) uses feature points 201 orfeature lines 202.

According, in some embodiments, correspondences between an initialskeletal model (or current skeletal model) and the output of the CNN(i.e., the output image data) may be determined using one or more activesearch techniques.

In accordance with a first technique, used in relation to feature points201, for one or more of the feature points 201, P_(n) ^(c)∈

^(c) an estimated location of the feature point 201 is determined withinthe output image data. Accordingly, for point based measurements, I_(i)^(P), this may be achieved by projecting a current estimated location ofthe feature point 201 (e.g., from the initial model or as a result ofusing the mapping process (see elsewhere herein)) into the frame ofreference of the camera (or other sensor) which captured the input imagedata—as described herein—or otherwise projected into a frame ofreference of the output image data. For example, a current parameterizedskeletal model may be projected into the frame of reference of theimaging sensor using one or more parameters (which may include, forexample, the intrinsic parameters, K, of the imaging sensor) and usingthe current estimate pose of the imaging sensor, {circumflex over (T)}₁.This projection may be into the image plane, Vin, for example.Accordingly, the expected location of one or more feature points 201 inthe output image data can be determined.

A search may then be performed to match each feature point 201 of themodel with the corresponding feature represented in the output imagedata.

Accordingly, a search region may be defined around the expected locationof the or each feature point 201 within the output image data. Thissearch region may be defined by a radius, r^(P), from the expectedlocation, for example. A pixel with the highest value above a threshold,λ^(P), within the search region is identified as corresponding with thatfeature point 201. If no pixel values exceed the threshold, then nocorrespondence is found. The process continues through all featurepoints 201.

In accordance with a second technique, used for feature lines 202, oneor more points, Q^(c)={q₁ ^(c), . . . , q_(m) ^(c)}, may be determinedalong the feature line 202,

^(c). This may be achieved by dividing the feature line 202 into two ormore sub-lines and taking the ends of those sub-lines as the one or morepoints (which may be referred to as line points, as they are pointsalong a feature line 202). The number of line points used for eachfeature line 202 may be dependent, in part, on the length of thatfeature line 202. So, for example, the first feature line 202 a may bedivided into s^(t) points, the second feature line 202 b into s^(h)points, and the third feature lines 202 c into s^(b) points for each ofthe third feature lines 202 c.

These line points, q_(m) ^(c), may then be projected into a frame ofreference of the output image data. For example, line points may beprojected into the frame of reference of the imaging sensor using one ormore parameters (which may include, for example, the intrinsicparameters, K, of the imaging sensor) and using the current estimatepose of the imaging sensor, {circumflex over (T)}₁, to form projectedline points, {circumflex over (q)}_(m) ^(I). This projection may be intothe image plane, {circumflex over (q)}_(n) ^(I), for example.

A search may then be performed to match each line point with thecorresponding point represented in the output image data.

Accordingly, a perpendicular line search may be used to identify thecorresponding points within the output image data. This may be achievedby, for example, projecting two end points of each feature line 202 intoa frame of reference of the output image data (e.g., into the frame ofreference of the imaging sensor, as described above). These two endpoints may be used to define a line therebetween in relation to theoutput image data (i.e., within its frame of reference). A directionperpendicular to this line (which may be a 2D line) may be a searchdirection for a line search.

In this search direction, starting at the projected feature line,samples may be taken of pixel values at predetermined distances from theprojected feature line, k^(L), and over a predetermined length, a^(L).The sampled pixel with the highest value above a threshold, λ^(L), istaken to correspond with the projected line point. If no pixel valuesexceed the threshold, then no correspondence is found. The processcontinues through all line points.

After the use of one or more of the active search techniques, a set ofcorrespondences has been determined with respect to the output imagedata and the skeletal model.

An image cost function can then be determined as:

$\begin{matrix}{e_{i}^{I} = {{\beta^{p}{\sum\limits_{n}^{N_{i}}\;{{{\hat{p}}_{n}^{I} - p_{n}^{I}}}}} + {\beta^{q}{\sum\limits_{m}^{M_{i}}\;{{{\hat{q}}_{m}^{I} - q_{m}^{I}}}}}}} & (5)\end{matrix}$

-   -   where Ni represents to the number of point correspondences in        the image frame i, and    -   Mi represents to the number of line correspondences in image        frame i.    -   The values β^(p) and β^(q) may be used to weight the different        types of correspondences.

Accordingly, the cost function, E, (equation (3)), can then be used tooptimize the pose graph. This may be achieved using any suitablemethodology, for example, the Gauss-Newton algorithm may be used, whichworks by linearizing the problem around the current best guess solution,finding the minimum and repeating until convergence.

The optimization technique may be an iterative technique. One or morefurther constraints may be used in relation to the optimization of thepose graph. For example, the relative movement or likely movement of theautonomous vehicle 1 with respect to the object 5 since the lastlocalization may be used to constrain and simplify the optimization. Theresult is a new or refined pose (i.e., location and/or orientation) ofthe autonomous vehicle 1 and/or the imaging sensor.

In accordance with some embodiments, therefore, there is provided a poseoptimization process to optimize an approximate pose of a sensor withrespect to an object 5 using sensor data. This process may includereceiving a model of the object 5. Sensor data may be received relatingto at least part of the object 5, wherein that sensor data may be in asensor frame of reference and may be associated with pose datarepresenting a pose of a sensor used to generate the sensor data. Theprocess may include projecting at least part of the model onto thesensor frame of reference. Using this projection, corresponding featurepoints and/or feature lines between the model and sensor data in thesensor frame of reference may be identified. Finally, the pose data maybe optimized based on the identified correspondence, using a locationdifference in the corresponding feature points and/or feature linesbetween the model and the sensor data in the sensor frame of reference.

The sensor data may be image data, for example, and this could be sensordata from a ranging sensor—which is configured to generate a sensor datarepresenting at least part of the object 5 in the form of a range of theobject 5 from the sensor which generates the sensor data.

In some embodiments, there is provided a mapping and/or localizationprocess including the aforementioned pose optimization process followedby optimizing at least one parameter of the model of the object 5,wherein the at least one parameter defines an estimated configuration ofthe object 5, and wherein optimizing the at least one parameter includesoptimizing the at least one parameter based on the correspondencebetween the feature points and/or feature lines and the optimized posedata, such that the optimized parameter represents the configuration ofthe object 5.

FIG. 17 shows the model projected onto image data from an autonomousvehicle 1, wherein the model is projected both before and afteroptimization—with the post-optimization model being better aligned withthe image data.

The pose (i.e., location and/or orientation) of the autonomous vehicle 1and/or the imaging sensor may be defined in terms of coordinates andorientation angles or quaternions. These may be with respect to a fixedpoint associated with the object 5 (such as the base or top of the windturbine) or may be with respect to a fixed geographical point and thedirection of gravity.

As discussed herein, the skeletal model may be a parameterized modelwhich includes feature lines 202 and/or feature points 201 (i.e., atleast one parameter defining a configuration of the object 5).

In order to parameterize the model, the initial information (e.g.,initial setup parameters) and/or the initial setup program may be usedin order to generate parameters for the skeletal model. These will,however, likely be relatively inaccurate but may, nevertheless, beacceptable for generation of the initial model (the initial model being,in this case, an initial skeletal model). As will be understood, theparameterized initial skeletal model may then be used in thelocalization process described herein.

In some embodiments, a mapping process is provided for the skeletalmodel and operated as part of the navigation program. The mappingprocess for a skeletal model used in some embodiments may use similarsteps to the aforementioned localization process.

In particular, the mapping process for the skeletal model may includereceipt of one or more images of at least part of the object 5 (e.g.,the wind turbine) and may include receipt of the skeletal model (whichmay be parameterized using the initial information (e.g., initial setupparameters), for example). The one or more images may—as describedherein—be images acquired by an imaging sensor of the autonomous vehicle1 or may have been acquired by an independent camera or sensor for whichthe pose is known). The or each image is then pass through the CNN 300which may be of identical form and operation to that described above inrelation to the localization process (and may be the same CNN 300).Accordingly, the or each image may be input image data as describedherein and this data may be associated with metadata as describedherein.

The output image data from the CNN 300 may then be used for a comparisonwith the current parameterized skeletal model (which may be the initialmodel) in order to refine and re-parameterize the model.

This process may be similar to that described in relation to thelocalization process in relation to the pose graph. In particular, thecurrent skeletal model may be projected onto the output image data andthen the searching techniques described herein for feature points 201and feature lines 202 may be implemented to determine correspondencebetween the current skeletal model and the output image data—see thelocalization process description for a more detailed explanation. Theparameters of the skeletal model may then be optimized using a suitableoptimization technique (such as the Gauss-Newton algorithm) in order todetermine new parameters for the skeletal model. The optimizationtechnique may be an iterative technique.

This re-parameterized skeletal model may then become the currentskeletal model, and may be used by the navigation program.

The re-parameterization of the skeletal model in this manner may occurone or more times during an operation of the autonomous vehicle 1. Insome embodiments, the mapping process and localization process for theskeletal model occur repeatedly in a substantially continuous manner. Insome embodiments, the localization process for the skeletal model occursrepeatedly in a substantially continuous manner but the mapping processmay occur less frequently—e.g., periodically or at predeterminedstage(s) of the operation of the autonomous vehicle 1.

As will be understood, the described mapping and/or localizationprocesses for skeletal models can operate using input image data whichrepresents just part of the object 5 but, in some embodiments, the inputimage data provides a complete view of the object 5 (from a particularangle).

The object configuration and object location may, therefore, form amodel of the object 5. The navigation program may be configured togenerate multiple such models using different combinations of sensordata and/or different mapping and/or localization processes.

The navigation program may be configured to use one or more of thesemodels (which are in-flight models) to update the initial model. Thismay include the combining of multiple in-flight models. The combinationof in-flight models may be performed using a number of differentmethods—such as by taking an average (e.g., mean) of the models, usingstandard deviations and thresholding, or the like.

The navigation program may be configured to generate an in-flight modelto replace the initial model for use in controlling the operation of thevehicle 1. This in-flight model may take into account one or more of thein-flight models which are generated as a result of the mapping and/orlocalization processes. In some embodiments, therefore, the replacementmodel may be referred to as the replacement model or the determinedin-flight model. In-flight models which are used to generate thereplacement model may be referred to as temporary or working in-flightmodels, for example.

The navigation program may be configured to exclude sensor data from oneor more sensors 171 due to, for example, a detected error with thesensor 171 or sensor data which is not within an expected predeterminedrange or within a specified range of other sensor data. The navigationprogram may be configured to perform a similar process at the in-flightmodel level—excluding one or more in-flight models which are generatedbased on sensor data from a sensor 171 in relation to which an error hasbeen detected, or based on a model which is not within a specified rangeof another in-flight model or models.

The processes of clustering the sensor data, analyzing the clusteredsensor data, and determining according to the mapping and localizationprocesses, may be undertaken in different ways. For example, clusters ofsensor data may each be approximated to an appropriate shape which maybe a 2D or a 3D shape depending on the form of the sensor data (such asa cylinder, ellipse, circle, or the like)—3D sensor data may berestricted to provide 2D sensor data in some embodiments or 3D sensordata may be used or the sensor data may be provided as 2D sensor data.The approximated shapes may then be analyzed and compared to the initialmodel. This may include, for example, analyzing a perimeter of theapproximated shapes compared to parts of the initial model or mayinclude analyzing centroids of the approximated shapes compared tocentroids of parts of the initial model.

In some embodiments, these processes may include the use of linearregression and/or iterative closest point approaches.

In some embodiments, the sensor data is in the form of a point cloud ormay be converted to a point cloud for processing according to themapping and/or localization processes.

Turning back to the mapping process, in some embodiments, the mappingprocess includes the navigation program using information from thesensor system 17 to generate one or more models (herein a real-time orin-flight model) of the object 5. The in-flight model or models may bedifferent from the model which may be used by the navigation program todetermine the flight plan initially (and/or may be the same nominalmodel but with one or more different parameters).

The navigation program may use information received from the sensorsystem 17 (i.e., the sensor data, which may include range information)which represents the object 5 and the position of the object 5 relativeto the vehicle 1 (e.g., relative to the inspection system 19).

The sensor data (which may be combined data from multiple sensors 171 ordata from a single sensor 171) may be separated by the navigationprogram into sensor data relating to different parts of the initialmodel. This may, as described below or otherwise, include clustering orsplitting the sensor data using the initial model to provide spatialconstraints.

The mapping process may include clustering or splitting the sensor datainto that relating to two or more different parts of the object 5. Thismay include splitting the sensor data into that data relating to a towerand that data relating to one or more blades, for example.

The splitting of the sensor data may be achieved in a number ofdifferent ways. For example, the sensor data (which may include rangedata and position data) may be used to generate a point cloudrepresentation of the object 5, a part thereof, and/or any other nearbyobjects within the effective range of the sensor or sensor 171 beingused.

Using a K-means approach or otherwise, the point cloud or other sensordata may be clustered—i.e., separated into the parts which represent oneor more parts of the initial model (e.g., the nominal model) of theobject 5.

The navigation program may then estimate a centroid or other referencepoint of at least one part of object 5 as represented in the clusteredsensor data—e.g., taking each part of the object 5 to be represented bya shape such as an ellipse.

The navigation program may then attempt one or more transformations tocompare the one or more parts of the object in the initial model of theobject 5 with the one or more parts from the clustered sensor data. Inthis respect, the navigation program may determine one or moretransformations of the centroid(s) and/or other reference point(s) withcorresponding points in the initial model.

The navigation program may be configured to determine a best fittransform for the object 5 as represented in the sensor data with theobject 5 as represented by the initial model 5. This may be achievedusing an iterative best fit method.

Similarly, for the localization process, the navigation program maydetermine a best fit transformation for the location of the object 5 asa whole (or a static part thereof such as the tower of a wind turbine)relative to the vehicle 1 (e.g., relative to the inspection system 19).Again, this may use an iterative best fit method.

The navigation program may then generate a transformed initial model ofthe object 5 which is based on the sensor data. This transformed initialmodel may be the in-flight model and may be a version of the nominalmodel used for initial model but with updated parameters.

The mapping process may be performed less frequently than thelocalization process if the configuration of the object 5 is unlikely tochange, for example. Alternatively, both processes can be performed onan ongoing and repeated basis.

In another version of the mapping process, a centroid of a first part ofthe object (e.g., the tower) is taken to be a center of the object 5. Inthis part of the process, a moving average may be used to smooth theestimation and accommodate false measurements. The navigation programmay be configured to ignore the height of parts of the object 5 togenerate a 2-D representation of the centroids of second parts (orsub-parts) of the object 5 (e.g., the blades). The navigation programmay perform a linear regression on these points to estimate of the angleof the second part of the object 5 with respect to the first part of theobject 5 (for example) using a gradient of the or each line generated asa result of the 2-D representation. The navigation program may beconfigured to cluster the centroids of the sensor data relating to thesecond part of the object 5 into 3 elements using a K-means approach.The navigation program may be configured to perform a linear regressionto each of the clusters of the sensor data relating to the second partsof the object 5. The navigation program may be configured to generate amean intersection point of the resulting lines representing the secondpart of the object 5 to determine an approximate the height of theobject 5. A gradient of each of the lines representing the second partof the object 5 can be used to estimate the position of the second partwith respect to the first part (e.g., blade position with respect to thetower). Accordingly, the configuration of the object 5 may be determinedand the initial model updated to generate the in-flight model.Localization may be performed as described above.

Iterative closest fit and linear regression approaches have beendiscussed above. However, in some embodiments, the mapping and/orlocalization processes may use other techniques in order to generate thein-flight model.

As will be appreciated, the model of the object 5 may be determined, atleast in part, based on the initial information (e.g., initial setupparameters), for example. As will be understood, the sensor data used inthis process may include sensor data generated by the or each sensor 171(which may include range information, e.g., from a LiDAR sensor, orother depth/range sensor as described herein).

This process may be repeated in-flight—resulting in refinement of thein-flight model (see FIG. 8 , for example, which shows an original modeloverlaid with a new representation of the blades of a turbine).

In some embodiments, the mapping and/or localization processes of thenavigation program may include using one or more such techniques togenerate the in-flight model. In embodiments, in which multipletechniques are used, then the results may be combined—which may be usingstandard deviations, an average (e.g., a mean), and may be a weightedcombination based on covariance or on sensor status for the or eachsensor 171 used to generate the data.

This may, in some embodiments, comprise the navigation program combiningsensor data and then using the combined sensor data to generate thein-flight model or using the sensor data to generate multiple in-flightmodels and then combining the in-flight models.

In some embodiments, a neural network or a particle filter or a Kalmanfilter is used to combine a plurality of in-flight models and/or sensordata. In some embodiments, the most recent sensor data is used togenerate the in-flight model. In some embodiments, the sensor data isweighted according to the sensor 171 which generated the sensor data.This may include, for example, changing the weighting in-flight duringoperation of the autonomous vehicle 1.

The mapping process is configured to generate the in-flight model whichis then used by the localization process, which positions that in-flightmodel within a frame of reference used by the autonomous vehicle 1 and,in particular, by the navigation program.

In some embodiments, the mapping and/or localization processes areundertaken using multiple in-flight models which are then combined—e.g.,using an average (e.g., a mean) or any other suitable method—todetermine the configuration and/or location of the autonomous vehicle 1.

Thus, in some embodiments, the in-flight model is an important aspect ofthe operation of the navigation program. In some embodiments, therefore,the navigation program may include, as part of the flight plan, one ormore transition locations which are intended to guide the autonomousvehicle 1 along an initial mapping segment of the flight plan. In thisinitial mapping segment, the autonomous vehicle 1 may be moved relativeto the object 5 (based on the model of the object used in the generationof the flight plan, i.e., an initial model) to allow the mapping and/orlocalization processes to be undertaken in order to generate thein-flight model. This initial mapping segment may include, for example,controlling the movement of the autonomous vehicle 1 to travel withrespect to at least a portion of the object 5 such that the at least aportion of the object 5 can be sensed by the sensor system 17—e.g.,within range of the one or more inspection sensors 171.

The initial mapping segment may, therefore, be configured to causemovement of the autonomous vehicle 1 along a predetermined length of theobject 5. In some embodiments, initial mapping segment may include, inaddition to the movement along a predetermined length of the object, afurther movement in the same direction. In particular, it will beunderstood that the predetermined length is determined based on themodel of the object 5 (before the in-flight model has been generated).If the actual object 5 extends beyond this predetermined length, thenthis may be useful information for use in generating the in-flightmodel. The further movement in the same direction may include a movementof an additional 20% compared to the predetermined length, for example.

In the case of the object 5 being a wind turbine, then the initialmapping segment may be a movement of the autonomous vehicle 1 along alength of the tower of the wind turbine—e.g., with the autonomousvehicle 1 positioned with respect to the wind turbine such that thesensor system 17 can sense the tower of the wind turbine. In someembodiments, the predetermined distance may be a substantial part of thelength of the tower. In some embodiments, the predetermined distance maybe an upper length of the tower. In some embodiments, the furthermovement is a movement above the top of the tower (based on the modelused). In some embodiments, the further movement would take theautonomous vehicle 1 to a height of 120% of the height of the tower(based on the model used). FIG. 11 shows an autonomous vehicle 1performing such a movement along a part of the length of the tower. FIG.10 shows an autonomous vehicle 1 approaching the top of the tower.

In some embodiments, therefore, the mapping process may occur after theautonomous vehicle 1 has been moved in accordance with the initialsegment mapping of the flight plan.

In some embodiments, the localization process includes the navigationprogram determining the relative location of the autonomous vehicle 1with respect to the in-flight model. In other words, the localizationprocess may include positioning the in-flight model within a frame ofreference of the autonomous vehicle 1.

In some embodiments, the localization process uses location information(i.e., sensor data) provided by the sensor system 17. The locationinformation may include general location information—which may includesensor data from a sensor 171 which is a radio navigation sensor. Thelocalization process may include an initial localization step in whichthe initial model (i.e., as used to generate the flight plan) ispositioned with a frame of reference to the autonomous vehicle 1 usingsuch general location information. In some embodiments, the localizationprocess continues during movement of the autonomous vehicle 1 along theinitial mapping segment of the flight plan. During this part, thelocation of the in-flight model within the frame of reference of theautonomous vehicle 1 is determined. During this stage, more specificlocation information may be used from one or more others of the sensors171—such as sensor data from a ranging sensor 171 as described herein.

The localization process may continue through all or part of themovement of the autonomous vehicle 1 along the flight path under thecontrol of the navigation program.

During the mapping and localization processes, the navigation program,adjusts the flight path so that this matches the original flight planbut now based on the in-flight model.

Accordingly, as will be appreciated, the flight plan is generated by thenavigation program based on an initial model (i.e., representation) ofthe object 5. This model may be used to determine the inspectionlocations and then the required sequence of locations (inspection andtransition) to define the flight path and plan. The model is configuredto allow an angular position of the inspection system 19 to bedetermined with respect to the object 5 to ensure proper inspect of theobject 5. Initial flight may be based on this model and flight plan.Then during flight, a different or refined model may be used to updatethe flight plan (i.e., the in-flight model). The requirements of eachmodel are different and the use of a different model in-flight reducesthe computational overheads associated with updating the original model,for example. The original model may be referred to as a first model, anoriginal model, or an initial model (the in-flight model may also bereferred to, therefore, as the second model).

As mentioned above, FIG. 12 depicts an example flow diagram of a processof generating an in-flight model from combining two models generatedusing different data, along with the initial model (which may be theoriginal model). This process may be implemented by the navigationprogram and may include combinations or more than two models (each ofwhich may be generated using different sensor data to that shown in thefigure). In some embodiments, such as that depicted in the diagram ofFIG. 12 , the in-flight model may be updated and new positioninformation for the vehicle 1 may also be generated—which may be donealternately, for example. FIG. 13 shows a similar flow diagram with thespecific example of the use of a skeletal model with mapping andlocalization processes as described herein.

The mapping and/or localization processes may occur in-flight and mayuse data which is not the most recent sensor data acquired. In someinstances, the mapping and/or localization processes may use historicdata from earlier in the same flight. In some embodiments, a flight maybe reviewed in post-flight analysis which includes completion of themapping and/or locations processes using sensor data from the flight.The results may then be used in future flights—e.g., to modify theoriginal or initial model.

The in-flight model (and the original model) may include representationsof one or more of a center location of the bottom of the object (e.g. atower of the wind turbine) relative to the initial position of theautonomous vehicle, a height of the object (e.g. a height of a center ofthe wind turbine blades) relative to the initial position of theautonomous vehicle, the rotation position (e.g. in degrees) of a firstpart of the object 5 with respect to another part (such as a first bladefrom a vertical position), a heading from the initial position of theobject 5 with respect to the initial position of the autonomous vehicle1, and a blade center location (e.g. a distance of a center of the towerto the theoretical point where the blades meet).

Embodiments of the navigation program may use one or more differentframes of reference during its execution. These frames of reference mayinclude frames of reference of a geographically universal nature (e.g.,longitude and latitude) or locally defined frames of reference withrespect to the autonomous vehicle 1 and/or the object 5 or a partthereof.

The operations and processes of the navigation program may generally beseparated into two parts—(i) location determining parts and (ii)inspection parts. Location determining parts include all of theprocesses which determine or refine the location and configuration ofthe object 5 with respect to the autonomous vehicle 1. The Inspectionparts include the parts which are involved in inspection of the object 5(including the control of the movement of the autonomous vehicle 1). Theperformance of these parts of the navigation program may generally bereferred to as the “mission”. Aspects of the operation of the autonomousvehicle 1 which are not directly associated with the mission may includepreparatory processes (such as starting up the drive system 12, bootingup any of the computers described herein, performing initial pre-flightsafety checks, and performing any emergency post-mission or mid-missionactions. As will be understood, there may be pauses in the operation ofthe inspection parts of the operation of the navigation programfollowing instances of the operation of the location determining parts(e.g., as changes are identified).

The navigation program may include an in-flight safety process. Thein-flight safety process may be configured to execute whilst thenavigation program is controlling movement of the autonomous vehicle 1.The in-flight safety process may be configured to determine whetherthere are any reasons why the flight cannot continue. If any suchreasons are determined, then one or more emergency actions may beperformed.

The in-flight safety process may, therefore, check a plurality ofdifferent aspects of the operation of one or more of the navigationprogram itself, the autonomous vehicle 1, the base station 2, and theoperator interface device 4.

The in-flight safety process may include checking one or more of:

-   -   whether the sensor system 17 is operating correctly;    -   whether one or more of the sensors 171 is operating correctly        (e.g., one sensor 171 which indicates a position of the        autonomous vehicle 1 (e.g., with respect to the object 5) which        is more than a predetermined threshold from that indicated by        another sensor 171 (or an average of the position determined by        multiple other sensors 171) may trigger an emergency action);    -   whether the inspection system 19 is operating correctly;    -   whether one or more inspection sensors 191 is operating        correctly;    -   whether the power source 14 has sufficient power reserves to        complete the flight plan;    -   whether there is a fault from the power source 14;    -   whether the flight control system 15 is operating correctly;    -   whether the navigation system 16 is operating correctly;    -   whether the drive system 12 is operating correctly;    -   whether the visual guidance system 20 is operating correctly;        and    -   whether the communication system 18 is operating correctly        and/or whether communications are established with the base        station 2 and/or the operator interface device 4, and/or the        remote server 3.

On detecting a potential fault, the in-flight safety process isconfigured to determine if the fault will impact the completion or safecompletion of the flight plan, and/or will impact the ability of theautonomous vehicle 1 to return to the home location, and/or will impactthe ability of the autonomous vehicle 1 to land at another location(i.e., other than the home location).

The in-flight safety process may be configured to take a course ofaction based on this assessment. If the flight plan can be completeddespite the detected fault (and the fault does not impact the operationof the inspection system 19), then the in-flight safety process may notinterrupt the operation of the rest of the navigation program. In somesuch embodiments, an alert may be sent for display on the operatorinterface device 4 to notify the operator of the fault.

If the flight plan cannot be completed in light of the fault, then thein-flight safety process may be configured to determine a new flightplant to return the autonomous vehicle 1 to the home location(“RTL”—return to launch). This process may include determining one ormore transition locations. In some embodiments, the in-flight safetyprocess may determine whether the new flight plan (an emergency returnto home flight plan) is a safe flight path for the autonomous vehicle1—i.e., that the path is not likely to cause the autonomous vehicle 1 toimpact the object 5 or another object (other than the object 5). If theflight path is safe to complete, then the in-flight safety process maycause the autonomous vehicle 1 to travel along the emergency return tohome flight plan to land at the home location. In some such embodiments,an alert may be sent for display on the operator interface device 4 tonotify the operator of the action taken.

If the in-flight safety process cannot determine a safe flight plan toreturn the autonomous vehicle 1 to the home location and/or the processdetermines that the autonomous vehicle 1 cannot reach the home location(e.g., due to inadequate power), then the in-flight safety process maydetermine a new flight path to a different location (other than the homelocation). This process may include determining one or more transitionlocations. In some embodiments, the in-flight safety process maydetermine whether the new flight plan (an emergency landing flight plan)is a safe flight path for the autonomous vehicle 1—i.e., that the pathis not likely to cause the autonomous vehicle 1 to impact the object 5or another object (other than the object 5). If the flight path is safeto complete, then the in-flight safety process may cause the autonomousvehicle 1 to travel along the emergency landing flight plan to land theautonomous vehicle 1 at the other location. In some such embodiments, analert may be sent for display on the operator interface device 4 tonotify the operator of the action taken.

If the in-flight safety process cannot cause the landing, or cannotcause the safe landing of, the autonomous vehicle 1 at another location,then the in-flight safety process may send an alert for display on theoperator interface device 4 to notify the operator that the operatorshould use the operator interface device 4 to take manual control of theflight of the autonomous vehicle 1. The operator may input one or morecommands via the interface 42 which are sent to the autonomous vehicle 1(directly or via the base station 2) and which are then used to controlthe operation of the drive system 12 to control the movement of theautonomous vehicle 1.

In accordance with some embodiments, the operator interface device 4 isconfigured to receive a command from the operator (e.g., via theinterface 42) indicating that the operator will take manual control ofthe autonomous vehicle 1—in much the same manner as described above butwithout requiring the detection of a fault.

As will be appreciated, the references to a flight plan or flight pathare references to a navigation plan or path and are likewise referencesto an inspection plan or inspection path.

The operations of the navigation program of some embodiments are shownin FIG. 9 for ease of reference—which shows a state flow.

As will be understood, the navigation program provides an updatedin-flight model which is used during flight of the autonomous vehicle 1to control the flight of the vehicle 1. The flight plan itself may notsubstantially change in some embodiments but may be shifted in space asthe actual location of the object 5 is refined within the model.However, as will be appreciated, this shifting of the flight plan doesconstitute a change in the flight plan as well as in the model duringflight.

The autonomous vehicle 1, as will be understood, follows a flight plan(which may be updated in-flight) which causes the autonomous vehicle 1to move between at least two inspection locations (a first inspectionlocation and a second inspection location). At each inspection location,the inspection system 19 is used to inspect the object 5.

The inspection system computer 192 may be provided with informationconcerning the inspection locations. This information may include adistance between the inspection system 19 (and, in particular, the oreach inspection sensor 191 such as a camera 191 a). The information is,therefore, referred to as inspection location data.

The inspection system computer 192 may be configured to control theoperation of the one or more inspection sensors 191 (such as a camera191 a) in order to ensure that the object 5 is inspected by the or eachinspection sensor 191 accurately. In some embodiments, the inspectionsystem computer 192 may be configured to control the operation of theone or more inspection sensors 191 based on the inspection locationdata.

In some embodiments, this process includes the inspection systemcomputer 192 determining one or more inspection system parameters basedon the inspection location data. In some instances, however, theinspection location data includes the one or more inspection systemparameters. The one or more inspection system parameters includeparameters which are used to configure at least one of the one or moreinspection sensors 191. The or each inspection system parameter mayinclude, for example, (e.g., in the case of a camera 191 a) focussetting (such as a focal distance), a shutter speed, an aperture size,and the like.

As will be appreciated, the inspection location data may vary from oneinspection location to another. In particular, the inspection locationdata for the first inspection location may be different to theinspection location data for the second inspection location. Likewise,there may be more inspection locations and each may have differentinspection location data to another inspection location.

The inspection locations form a sequence in accordance with the flightplan (and may be interspersed with transition locations, as describedherein).

The inspection system computer 192 may be configured to control theoperation of the or each inspection sensor 191 using the inspectionlocation data (and the or each inspection system parameter) as theautonomous vehicle 1 moves between inspection locations. In particular,as the autonomous vehicle 1 moves towards an inspection location, theinspection system computer 192 may control the or each inspection sensor191 such that the or each inspection sensor 191 is configured to performan inspection operation on the object 5 when the autonomous vehicle 1arrives at that inspection location. In other words, the inspectionsystem computer 192 may have configured the or each inspection sensor191 to inspect the object 5 at the first inspection location using firstinspection location data (and one or more first inspection systemparameters). After the inspection system 19 has been used in a firstinspection operation at the first inspection location, the autonomousvehicle 1 commences movement towards the second inspection location(e.g., directly or via one or more transition locations (“towards” inthis instance being in accordance with the flight plan determinedsequence)). After the inspection system 19 has completed the firstinspection operation, the inspection system computer 192 may beconfigured to control the operation of the or each inspection sensor 191to configure the or each inspection sensor 191 using the secondinspection location data (and one or more second inspection systemparameters). As or before the autonomous vehicle 1 arrives at the secondinspection location, the or each inspection sensor 191 is, therefore,configured for a second inspection operation at the second inspectionlocation. This process reduces the amount of time the autonomous vehicle1 must remain at each inspection location.

In some embodiments, the or each inspection system parameter may berefined at the inspection location based one or more sensed parametersat the inspection location (e.g., sensed by one or more of theinspection sensors 191 and/or one or more sensors 171 of the sensorsystem 17). Thus, for example, a more precise (or actual) distance ofthe autonomous vehicle 1 (and so the inspection sensor 19) from theobject 5 may be determined at the inspection location. As will beappreciated, as this may be fine tuning of the configuration of the oreach inspection sensor 191, the majority of any configuration may haveoccurred as the autonomous vehicle 1 was still moving. Thus, thisprocess may still reduce the amount of time the autonomous vehicle 1must remain at each inspection location.

In some embodiments, the inspection system 19 is configured to store theinspection data (under the control of the inspection system computer192) in the storage medium 192 a after the or each inspection operation.In some embodiments, the inspection system 19 is configured to send theinspection data to the communication system 18 (e.g., to one or more ofthe communication interfaces 181) for transmission from the autonomousvehicle 1 (e.g., to the base station 2, the remote server 3, and/or theoperator interface device 4).

The inspection system computer 192 may be configured to determinewhether to send the inspection data (e.g., after an inspection operationat an inspection location) to the communication system 18 based at leastin part on the flight plan. In some embodiments, the inspection systemcomputer 192 is configured to determine whether to send the inspectiondata based on one or more of the expected length of time to travel tothe next inspection location, the volume of inspection data collated forthe last inspection operation (and/or any other inspection operationsfor which the inspection data has not yet been sent), the transmissionrate available using the communication system 18 (and, e.g., one or moreof the communication interfaces 181 for use in the sending of theinspection data), and the availability or signal strength of the one ormore communication interfaces 181 for use in sending of the inspectiondata. Accordingly, the inspection data 19 may be sent from theautonomous vehicle 1 during appropriate portions of the flight of theautonomous vehicle 1. This may make more efficient use of availableresources on the autonomous vehicle 1 and, for example, decrease thetime which may otherwise be required to extract the inspection data fromthe autonomous vehicle 1.

In some embodiments, the communication system 18 is configured todetermine if and when to send the inspection data from the autonomousvehicle 1—in the same manner as described above—and may, therefore, beconfigured to request inspection data from the inspection system 19 whenrequired.

As will be appreciated, therefore, in some embodiments, the inspectionsystem 19 is configured to perform the following operations:

-   -   a configuration operation (to configure the or each inspection        sensor 191 to perform the inspection operation);    -   the inspection operation (to capture the inspection data); and    -   a send/store operation (to send and/or store the inspection        data).

As will also be appreciated, the configuration operation may beundertaken (at least partially, if not entirely) during flight to aninspection location). The inspection operation occurs at the inspectionlocation and the send/store operation may occur at least partiallyduring flight to another inspection or transition location. In someembodiments, the send/store operation occurs at least partially whilstthe autonomous vehicle 1 is at an inspection location (and may,therefore, overlap with the inspection operation in terms of when theyare performed).

The operations above in relation to the inspection system computer 192may be undertaken in accordance with an inspection system program whichmay be configured for execution by the inspection system computer 192.This program need not, in some embodiments, be executed by theinspection system computer 192 but could be executed by another computer(e.g., such as one of the other computers described herein).

The autonomous vehicle 1 may be a rotorcraft such as a helicopter whichmay be a multirotor helicopter (e.g., helicopter having two or morerotors providing lift and maneuvering forces). The autonomous vehicle 1may be a fixed wing aircraft (which may be a glider or a poweredaircraft). The autonomous vehicle 1 may be an aerostat (such as anairship). The autonomous vehicle 1 may be a ground vehicle whichincludes at least one ground engaging member—such as a wheel ortrack—and which is configured to be driven across a ground surface. Theautonomous vehicle 1 may be a watercraft, such as a boat or submarine.The autonomous vehicle may be an amphibious vehicle, for example, whichis configured to travel through water and across a ground surface—suchas a hovercraft, for example. As will be appreciated, embodiments of thepresent invention are applicable to different types of vehicle and theabove described embodiments could be implemented in relation to any suchautonomous vehicle 1.

Whilst embodiments have been described in relation to autonomousvehicles 1, it will be appreciated that the processes of mapping andlocalization can be applied to other fields (i.e., other than navigatingan autonomous vehicle 1). For example, the mapping and localizationprocesses described in relation to the skeletal model may be used inrelation to a mobile device which includes an imaging sensor configuredto generate image data. The device may further include a pose sensorconfigured to determine a pose of the device and/or camera (or othersensor)—this may include one or more accelerometers and/or radionavigation sensors, for example. The mobile device may, therefore, beconfigured to determine its pose using input image data captured by theimaging sensor and using a model of the object, a part thereof, and/orat least a part of the local environment. In order to achieve this, thedevice may execute the mapping and/or localization processes forskeletal models as described herein.

The mobile device may be a portable device which is configured to becarried by a human user, for example. This may include a mobiletelephone, a tablet computer, an augmented reality headset (or handset),or the like.

In relation to an augmented reality headset (or handset), the device maybe configured to overlay information on a screen of the device throughwhich the user can view the local environment and/or at least part theobject. The positioning of the overlaid information and/or theinformation content may be determined, at least in part, by the locationof the device and the environment or object visible to the user throughthe screen.

In some embodiments, the overlaid information may include navigationalinformation directing the user to travel in a particular direction basedon the location of the device.

When used in this specification and claims, the terms “comprises” and“comprising” and variations thereof mean that the specified features,steps or integers are included. The terms are not to be interpreted toexclude the presence of other features, steps or components.

The features disclosed in the foregoing description, or the followingclaims, or the accompanying drawings, expressed in their specific formsor in terms of a means for performing the disclosed function, or amethod or process for attaining the disclosed result, as appropriate,may, separately, or in any combination of such features, be utilized forrealizing the invention in diverse forms thereof.

Although certain example embodiments of the invention have beendescribed, the scope of the appended claims is not intended to belimited solely to these embodiments. The claims are to be construedliterally, purposively, and/or to encompass equivalents.

Representative features are set out in the following clauses (labelledas “aspects”), which stand alone or may be combined, in any combination,with one or more features disclosed in the text and/or drawings of thespecification.

Aspects

-   -   A1. A navigation program for an autonomous vehicle, the        navigation program configured to:        -   determine an initial inspection plan for an autonomous            vehicle to inspect an object based on an initial model of            the object;        -   control movement of the autonomous vehicle in accordance            with the initial inspection plan;        -   receive sensor data from a sensor system during movement of            the autonomous vehicle in accordance with the initial            inspection plan;        -   generate an in-flight model of the object based on the            received sensor data; and        -   generate an updated inspection plan using the in-flight            model and the initial inspection plan.    -   A2. A navigation program according to aspect A1, further        configured to:        -   receive an initial model of the object to be inspected by            the autonomous vehicle;        -   identify an inspection target associated with the initial            model of the object; and        -   determine an inspection location for the autonomous vehicle            from which inspection target is inspectable by an inspection            system of the autonomous vehicle, wherein the initial model            includes one or more convex shapes representing the object            and the initial inspection plan includes the inspection            location.    -   A3. A navigation program according to aspect A1 or A2, wherein        the received sensor data is generated by a ranging sensor of the        sensor system.    -   A4. A navigation program according to aspect A3, wherein the        ranging sensor is a LiDAR sensor.    -   A5. A navigation program according to any preceding aspect,        further configured to generate the in-flight model at least in        part by clustering the sensor data and associating one or more        geometric shapes with the clustered sensor data.    -   A6. A navigation program according to any preceding aspect,        further configured to generate the in-flight model at least in        part by use of an iterative closest point technique using sensor        data in the form of a point cloud.    -   A7. A navigation program according to any preceding aspect,        further configured to merge a plurality of in-flight models to        arrive at the in-flight model.    -   A8. A navigation method for an autonomous vehicle, the        navigation method including:        -   determining an initial inspection plan for an autonomous            vehicle to inspect an object based on an initial model of            the object;        -   controlling movement of the autonomous vehicle in accordance            with the initial inspection plan;        -   receiving sensor data from a sensor system during movement            of the autonomous vehicle in accordance with the initial            inspection plan;        -   generating an in-flight model of the object based on the            received sensor data; and        -   generating an updated inspection plan using the in-flight            model and the initial inspection plan.    -   A9. A navigation method according to aspect A8, further        including:        -   receiving an initial model of the object to be inspected by            the autonomous vehicle;        -   identifying an inspection target associated with the initial            model of the object; and        -   determining an inspection location for the autonomous            vehicle from which inspection target is inspectable by an            inspection system of the autonomous vehicle, wherein the            initial model includes one or more convex shapes            representing the object and the initial inspection plan            includes the inspection location.    -   A10. A navigation method according to aspect A8 or A9, wherein        the received sensor data is generated by a ranging sensor of the        sensor system.    -   A11. A navigation method according to aspect A10, wherein the        ranging sensor is a LiDAR sensor.    -   A12. A navigation method according to any of aspects A8 to A11,        further including generating the in-flight model at least in        part by clustering the sensor data and associating one or more        geometric shapes with the clustered sensor data.    -   A13. A navigation method according to any of aspects A8 to A12,        further including generating the in-flight model at least in        part by use of an iterative closest point technique using sensor        data in the form of a point cloud.    -   A14. A navigation method according to any of aspects A8 to A13,        further including merging a plurality of in-flight models to        arrive at the in-flight model.    -   A15. A navigation system for an autonomous vehicle, the        navigation system configured to:        -   determine an initial inspection plan for an autonomous            vehicle to inspect an object based on an initial model of            the object;        -   control movement of the autonomous vehicle in accordance            with the initial inspection plan;        -   receive sensor data from a sensor system during movement of            the autonomous vehicle in accordance with the initial            inspection plan;        -   generate an in-flight model of the object based on the            received sensor data; and        -   generate an updated inspection plan using the in-flight            model and the initial inspection plan.    -   A16. A navigation system according to aspect A15, further        configured to:        -   receive an initial model of the object to be inspected by            the autonomous vehicle;        -   identify an inspection target associated with the initial            model of the object; and        -   determine an inspection location for the autonomous vehicle            from which inspection target is inspectable by an inspection            system of the autonomous vehicle, wherein the initial model            includes one or more convex shapes representing the object            and the initial inspection plan includes the inspection            location.    -   A17. A navigation system according to aspect A15 or A16, wherein        the received sensor data is generated by a ranging sensor of the        sensor system.    -   A18. A navigation system according to aspect A17, wherein the        ranging sensor is a LiDAR sensor.    -   A19. A navigation system according to any of aspects A15 to A18,        further configured to generate the in-flight model at least in        part by clustering the sensor data and associating one or more        geometric shapes with the clustered sensor data.    -   A20. A navigation system according to any of aspects A15 to A19,        further configured to generate the in-flight model at least in        part by use of an iterative closest point technique using sensor        data in the form of a point cloud.    -   A21. A navigation system according to any of aspects A15 to A20,        further configured to merge a plurality of in-flight models to        arrive at the in-flight model.    -   A22. An autonomous vehicle including a navigation program        according to any of aspects A1 to A7 or a navigation system        according to any of aspects A15 to A21.    -   B1. An inspection system program for an autonomous vehicle with        an inspection system for use in inspecting an object, the        inspection system program configured to:        -   receive an inspection plan indicating at least two            inspection locations and inspection location data associated            with each inspection location; and        -   control operation of the inspection system to configure one            or more inspection sensors of the inspection system based on            the inspection location data as the autonomous vehicle moves            between the at least two inspection locations, such that the            or each inspection sensor is configured to inspect the            object at each inspection location.    -   B2. An inspection method for an autonomous vehicle with an        inspection system for use in inspecting an object, the        inspection method including:        -   receiving an inspection plan indicating at least two            inspection locations and inspection location data associated            with each inspection location;        -   controlling operation of the inspection system to configure            one or more inspection sensors of the inspection system            based on the inspection location data as the autonomous            vehicle moves between the at least two inspection locations,            such that the or each inspection sensor is configured to            inspect the object at each inspection location.    -   B3. An inspection system for an autonomous vehicle with an        inspection system for use in inspecting an object, the        inspection system configured to:        -   receive an inspection plan indicating at least two            inspection locations and inspection location data associated            with each inspection location; and        -   control operation of the inspection system to configure one            or more inspection sensors of the inspection system based on            the inspection location data as the autonomous vehicle moves            between the at least two inspection locations, such that the            or each inspection sensor is configured to inspect the            object at each inspection location.    -   B4. An autonomous vehicle including an inspection system program        according to aspect B1 or an inspection system according to        aspect B3.    -   C1. A navigation program for an autonomous vehicle with an        inspection system for use in inspecting an object, the        navigation program configured to:        -   receive a nominal model of the object;        -   receive one or more parameters associated with the nominal            model to define an initial model of the object;        -   provide navigation instructions to control the operation of            the vehicle;        -   receive sensor data representing at least part of the            object;        -   analyze the sensor data to determine an in-flight model by            adjusting one or more parameters associated with the nominal            model.    -   C2. A navigation method for an autonomous vehicle with an        inspection system for use in inspecting an object, the        navigation method including:        -   receiving a nominal model of the object;        -   receiving one or more parameters associated with the nominal            model to define an initial model of the object;        -   providing navigation instructions to control the operation            of the vehicle;        -   receiving sensor data representing at least part of the            object;        -   analyzing the sensor data to determine an in-flight model by            adjusting one or more parameters associated with the nominal            model.    -   C3. A navigation system for an autonomous vehicle with an        inspection system for use in inspecting an object, the        navigation system configured to:        -   receive a nominal model of the object;        -   receive one or more parameters associated with the nominal            model to define an initial model of the object;        -   provide navigation instructions to control the operation of            the vehicle;        -   receive sensor data representing at least part of the            object;        -   analyze the sensor data to determine an in-flight model by            adjusting one or more parameters associated with the nominal            model.    -   C4. An autonomous vehicle including a navigation program        according to aspect C1 or a navigation system according to        aspect C3.    -   D1. A navigation program for an autonomous vehicle with an        inspection system for use in inspecting an object, the        navigation program configured to:        -   receive sensor data from a plurality of different sensors,            the sensor data representing at least part of the object;        -   cluster the sensor data based on an initial model of the            object;        -   determine the relative position of one or more parts of the            object with respect to each other or determine a position of            the object with respect to the vehicle; and        -   update the initial model using the determined position or            determined relative position.    -   D2. A navigation method for an autonomous vehicle with an        inspection system for use in inspecting an object, the        navigation method including:        -   receiving sensor data from a plurality of different sensors,            the sensor data representing at least part of the object;        -   clustering the sensor data based on an initial model of the            object;        -   determining the relative position of one or more parts of            the object with respect to each other or determine a            position of the object with respect to the vehicle; and        -   updating the initial model using the determined position or            determined relative position.    -   D3. A navigation system for an autonomous vehicle with an        inspection system for use in inspecting an object, the        navigation system configured to:        -   receive sensor data from a plurality of different sensors,            the sensor data representing at least part of the object;        -   cluster the sensor data based on an initial model of the            object;        -   determine the relative position of one or more parts of the            object with respect to each other or determine a position of            the object with respect to the vehicle; and        -   update the initial model using the determined position or            determined relative position.    -   D4. An autonomous vehicle including a navigation program        according to aspect D1 or a navigation system according to        aspect D3.    -   E1. A navigation program for an autonomous vehicle with an        inspection system for use in inspecting an object, the        navigation program configured to:        -   use an initial model of the object to instruct the movement            of the vehicle with respect to the object;        -   generate a plurality of in-flight models of the object using            sensor data from a plurality of different sensors collected            during movement of the vehicle with respect to the object;        -   combine the plurality of in-flight models; and        -   update the initial model using the combined plurality of            in-flight models.    -   E2. A navigation method for an autonomous vehicle with an        inspection system for use in inspecting an object, the        navigation method including:        -   using an initial model of the object to instruct the            movement of the vehicle with respect to the object;        -   generating a plurality of in-flight models of the object            using sensor data from a plurality of different sensors            collected during movement of the vehicle with respect to the            object;        -   combining the plurality of in-flight models; and        -   updating the initial model using the combined plurality of            in-flight models.    -   E3. A navigation system for an autonomous vehicle with an        inspection system for use in inspecting an object, the        navigation system configured to:        -   use an initial model of the object to instruct the movement            of the vehicle with respect to the object;        -   generate a plurality of in-flight models of the object using            sensor data from a plurality of different sensors collected            during movement of the vehicle with respect to the object;        -   combine the plurality of in-flight models; and        -   update the initial model using the combined plurality of            in-flight models.    -   E4. An autonomous vehicle including a navigation program        according to aspect E1 or a navigation system according to        aspect E3.    -   F1. A navigation program for an autonomous vehicle, the        navigation program configured to:        -   receive an initial model of an object to be inspected by the            autonomous vehicle;        -   identify an inspection target associated with the initial            model of the object; and        -   determine an inspection location for the autonomous vehicle            from which inspection target is inspectable by an inspection            system of the autonomous vehicle, wherein the initial model            includes one or more shapes representing the object which            permit a projection from the inspection location to the            inspection target to be determined.    -   F2. A navigation program according to aspect F1, wherein the        representation of the object in the initial model includes one        or more convex shapes.    -   F3. A navigation program according to aspect F1 or F2, further        configured to determine a plurality of inspection locations from        which the inspection target is inspectable.    -   F4. A navigation program according to any preceding aspect F1 to        F3, further configured to determine one or more inspection        locations for the autonomous vehicle from which one or more        further inspection targets are inspectable by the inspection        system.    -   F5. A navigation program according to aspect F4, further        configured to determine a plurality of inspection plans in which        each inspection target is inspected, optionally further        configured to check that each of the inspection plans does not        violate one or more constraints, and/or optionally further        configured to alter one or more of the inspection plans.    -   F6. A navigation program according to aspect F5, further        configured to cost the inspection plans in relation to a cost        criterion to determine a lowest cost inspection plan.    -   F7. A navigation program according to any preceding aspect,        further configured to:        -   determine an initial inspection plan for an autonomous            vehicle to inspect an object based on the initial model of            the object, the initial inspection plan including the            inspection location;        -   control movement of the autonomous vehicle in accordance            with the initial inspection plan;        -   receive sensor data from a sensor system during movement of            the autonomous vehicle in accordance with the initial            inspection plan;        -   generate an in-flight model of the object based on the            received sensor data; and        -   generate an updated inspection plan using the in-flight            model and the initial inspection plan.    -   F8. A navigation method for an autonomous vehicle, the        navigation method including:        -   receiving an initial model of an object to be inspected by            the autonomous vehicle;        -   identifying an inspection target associated with the initial            model of the object; and        -   determining an inspection location for the autonomous            vehicle from which inspection target is inspectable by an            inspection system of the autonomous vehicle, wherein the            initial model includes one or more convex shapes            representing the object.    -   F9. A navigation method according to aspect F8, wherein the        representation of the object in the initial model includes only        convex shapes.    -   F10. A navigation method according to aspect F8 or F9, further        including determining a plurality of inspection locations from        which the inspection target is inspectable.    -   F11. A navigation method according to any of aspects 8 to 10,        further including determining one or more inspection locations        for the autonomous vehicle from which one or more further        inspection targets are inspectable by the inspection system.    -   F12. A navigation method according to aspect F11, further        including determining a plurality of inspection plans in which        each inspection target is inspected, optionally further        including checking that each of the inspection plans does not        violate one or more constraints, and/or optionally further        including altering one or more of the inspection plans.    -   F13. A navigation method according to aspect F12, further        including costing the inspection plans in relation to a cost        criterion to determine a lowest cost inspection plan.    -   F14. A navigation method according to any of aspects F8 to F13,        further including:        -   determining an initial inspection plan for an autonomous            vehicle to inspect an object based on the initial model of            the object, the initial inspection plan including the            inspection location;        -   controlling movement of the autonomous vehicle in accordance            with the initial inspection plan;        -   receiving sensor data from a sensor system during movement            of the autonomous vehicle in accordance with the initial            inspection plan;        -   generating an in-flight model of the object based on the            received sensor data; and        -   generating an updated inspection plan using the in-flight            model and the initial inspection plan.    -   F15. A navigation system for an autonomous vehicle, the        navigation system configured to:        -   receive an initial model of an object to be inspected by the            autonomous vehicle;        -   identify an inspection target associated with the initial            model of the object; and        -   determine an inspection location for the autonomous vehicle            from which inspection target is inspectable by an inspection            system of the autonomous vehicle, wherein the initial model            includes one or more shapes representing the object which            permit a projection from the inspection location to the            inspection target to be determined.    -   F16. A navigation system according to aspect F15, wherein the        representation of the object in the initial model includes one        or more convex shapes.    -   F17. A navigation system according to aspect F15 or F16, further        configured to determine a plurality of inspection locations from        which the inspection target is inspectable.    -   F18. A navigation system according to any of aspects F15 to F17,        further configured to determine one or more inspection locations        for the autonomous vehicle from which one or more further        inspection targets are inspectable by the inspection system.    -   F19. A navigation system according to aspect F18, further        configured to determine a plurality of inspection plans in which        each inspection target is inspected.    -   F20. A navigation system according to aspect F19, further        configured to check that each of the inspection plans does not        violate one or more constraints.    -   F21. A navigation system according to aspect F20, further        configured to alter one or more of the inspection plans.    -   F22. A navigation system according to aspect F20 or F21, further        configured to cost the inspection plans in relation to a cost        criterion to determine a lowest cost inspection plan.    -   F23. A navigation system according to any of aspects F15 to F22,        further configured to:        -   determine an initial inspection plan for an autonomous            vehicle to inspect an object based on the initial model of            the object, the initial inspection plan including the            inspection location;        -   control movement of the autonomous vehicle in accordance            with the initial inspection plan;        -   receive sensor data from a sensor system during movement of            the autonomous vehicle in accordance with the initial            inspection plan;        -   generate an in-flight model of the object based on the            received sensor data; and        -   generate an updated inspection plan using the in-flight            model and the initial inspection plan.    -   F24. An autonomous vehicle including a navigation program        according to any of aspects F1 to F7 or a navigation system        according to any of aspects F15 to F23.    -   G1. A pose optimization process to optimize an approximate pose        of a sensor with respect to an object using sensor data, the        process comprising:        -   receiving a model of an object;        -   receiving sensor data relating to at least part of the            object, the sensor data being in a sensor frame of            reference, the sensor data being associated with pose data            representing a pose of a sensor used to generate the sensor            data;        -   projecting at least part of the model onto the sensor frame            of reference;        -   identifying corresponding feature points and/or feature            lines between the model and sensor data in the sensor frame            of reference; and        -   optimizing the pose data based on the identified            correspondence, using a location difference in the            corresponding feature points and/or feature lines between            the model and the sensor data in the sensor frame of            reference.    -   G2. A mapping and/or localization process including:        -   a pose optimization process according to aspect G1; and        -   optimizing at least one parameter of the model of the            object, wherein the at least one parameter defines an            estimated configuration of the object, and wherein            optimizing the at least one parameter includes optimizing            the at least one parameter based on the correspondence            between the feature points and/or feature lines and the            optimized pose data, such that the optimized parameter            represents the configuration of the object.    -   G3. A mapping and/or localization process according to aspect        G2, wherein the optimization of the at least one parameter and        the optimizing of the data representing the pose of the sensor        occur substantially simultaneously.    -   G4. A mapping and/or localization process according to aspect G2        or G3, further including processing the sensor data prior to        projecting at least part of the mode into the sensor frame of        reference, wherein processing the sensor data comprises passing        the sensor data through a convolutional neural network with a        convolutional-deconvolutional architecture to generate output        sensor data, and wherein identifying corresponding feature        points and/or feature lines between the model and sensor data in        the sensor frame of reference includes identifying corresponding        feature points and/or feature lines using the model and the        output sensor data.    -   G5. A mapping and/or localization process according to aspect        G4, wherein the output sensor data represents, at least in part,        a skeletal representation of the object.    -   G6. A mapping and/or localization process according to any of        aspects G2 to G5, wherein the optimization of the at least one        parameter and the optimizing of the data representing the pose        of the sensor are iterative optimization processes using a cost        function.    -   G7. A mapping and/or localization process according to any of        aspects G2 to G6, wherein the sensor data relating to at least        part of the object is multi-channel sensor data and includes a        projection of the model onto the sensor frame of reference.    -   G8. A mapping and/or localization process according to any of        aspects G2 to G7, further including:        -   determining the at least one parameter defining the            estimated configuration of the object using a different            model of the object.    -   G9. A mapping and/or localization process according to any of        aspects G2 to G8, wherein identifying corresponding feature        points and/or feature lines between the model and sensor data in        the sensor frame of reference includes defining a search region        around a feature point identified in the model and searching the        search region.    -   G10. A mapping and/or localization process according to any of        aspects G2 to G9, wherein identifying corresponding feature        points and/or feature lines between the model and sensor data in        the sensor frame of reference includes defining one or more line        points along a feature line identified in the model, defining        search line and direction, and searching in accordance with the        search line and search direction for one or more respective        corresponding line points.    -   G11. A mapping and/or localization process according to any of        aspects G2 to G10, wherein the sensor data includes image data.    -   G12. A pose optimization program stored as computer executable        instructions on a computer readable medium which, when executed,        cause the performance of the process of aspect G1.    -   G13. A mapping and/or localization program stored as computer        executable instructions on a computer readable medium which,        when executed, cause the performance of the process of any of        aspects G2 to G11.    -   G14. A navigation process for an autonomous vehicle, the        navigation process including:        -   performing a process according to any of aspects G1 to G11;            and        -   defining or modifying a navigation plan using the optimized            pose data.    -   G15. A navigation process according to aspect G14, wherein using        the optimized pose data includes the mapping and/or localization        process of any of aspects G2 to G11.    -   G16. A navigation program for an autonomous vehicle, the        navigation program stored as computer executable instructions on        a computer readable medium which, when executed, cause the        performance of the process of aspect G14 or G15.    -   G17. An autonomous vehicle including a program according to any        of aspects G12, G13, G15, or G16.

The invention claimed is:
 1. A pose optimization process to optimize anapproximate pose of a sensor with respect to an object using sensordata, the process comprising: receiving a model of an object; receivingsensor data relating to at least part of the object, the sensor databeing in a sensor frame of reference, the sensor data being associatedwith pose data representing a pose of a sensor used to generate thesensor data; projecting at least part of the model onto the sensor frameof reference; identifying corresponding feature points and/or featurelines between the model and sensor data in the sensor frame ofreference, wherein identifying corresponding feature points and/orfeature lines between the model and sensor data in the sensor frame ofreference includes defining a search region around a feature pointidentified in the model and searching the search region; and optimizingthe pose data based on the identified correspondence, using a locationdifference in the corresponding feature points and/or feature linesbetween the model and the sensor data in the sensor frame of reference.2. A non-transitory computer readable medium storing a pose optimizationprogram which, when executed, causes performance of the process ofclaim
 1. 3. A pose optimization process to optimize an approximate poseof a sensor with respect to an object using sensor data, the processcomprising: receiving a model of an object; receiving sensor datarelating to at least part of the object, the sensor data being in asensor frame of reference, the sensor data being associated with posedata representing a pose of a sensor used to generate the sensor data;processing the sensor data including passing the sensor data through aconvolutional neural network with a convolutional-deconvolutionalarchitecture to generate output sensor data; projecting, after theprocessing of the sensor data, at least part of the model onto thesensor frame of reference; identifying corresponding feature pointsand/or feature lines between the model and sensor data in the sensorframe of reference, wherein identifying corresponding feature pointsand/or feature lines between the model and sensor data in the sensorframe of reference includes identifying corresponding feature pointsand/or feature lines using the model and the output sensor data;optimizing the pose data based on the identified correspondence, using alocation difference in the corresponding feature points and/or featurelines between the model and the sensor data in the sensor frame ofreference; and optimizing at least one parameter of the model of theobject, wherein the at least one parameter defines an estimatedconfiguration of the object, and wherein optimizing the at least oneparameter includes optimizing the at least one parameter based on thecorrespondence between the feature points and/or feature lines and theoptimized pose data, such that the optimized parameter represents theconfiguration of the object.
 4. The pose optimization process accordingto claim 3, wherein the optimization of the at least one parameter andthe optimizing of the data representing the pose of the sensor occursubstantially simultaneously.
 5. The pose optimization process accordingto claim 3, wherein the output sensor data represents, at least in part,a skeletal representation of the object.
 6. The pose optimizationprocess according to claim 3, wherein the optimization of the at leastone parameter and the optimizing of the data representing the pose ofthe sensor are iterative optimization processes using a cost function.7. The pose optimization process according to claim 3, wherein thesensor data relating to at least part of the object is multi-channelsensor data and includes a projection of the model onto the sensor frameof reference.
 8. The pose optimization process according to claim 3,further including: determining the at least one parameter defining theestimated configuration of the object using a different model of theobject.
 9. The pose optimization process according to claim 3, whereinidentifying corresponding feature points and/or feature lines betweenthe model and sensor data in the sensor frame of reference includesdefining a search region around a feature point identified in the modeland searching the search region.
 10. The pose optimization processaccording to claim 3, wherein identifying corresponding feature pointsand/or feature lines between the model and sensor data in the sensorframe of reference includes defining one or more line points along afeature line identified in the model, defining search line anddirection, and searching in accordance with the search line and searchdirection for one or more respective corresponding line points.
 11. Thepose optimization process according to claim 3, wherein the sensor dataincludes image data.
 12. A non-transitory computer readable mediumstoring a pose optimization program which, when executed, causesperformance of the process of claim
 3. 13. The pose optimization processaccording to claim 3, wherein the configuration of the object includesthe relative position of parts of the object with respect to each other,wherein the parts of the object are moveable with respect to each other.14. A navigation process for an autonomous vehicle, the navigationprocess including: receiving a model of an object; receiving sensor datarelating to at least part of the object, the sensor data being in asensor frame of reference, the sensor data being associated with posedata representing a pose of a sensor used to generate the sensor data;projecting at least part of the model onto the sensor frame ofreference; identifying corresponding feature points and/or feature linesbetween the model and sensor data in the sensor frame of reference,wherein identifying corresponding feature points and/or feature linesbetween the model and sensor data in the sensor frame of referenceincludes defining one or more line points along a feature lineidentified in the model, defining search line and direction, andsearching in accordance with the search line and search direction forone or more respective corresponding line points; optimizing the posedata based on the identified correspondence, using a location differencein the corresponding feature points and/or feature lines between themodel and the sensor data in the sensor frame of reference; and definingor modifying a navigation plan using the optimized pose data.
 15. Anon-transitory computer readable medium storing a navigation program foran autonomous vehicle which, when executed, causes performance of theprocess of claim 14.